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Chapter  I:  Introduction 


1.1  Overview 

Efforts  to  examine  consumers'  preferences  regarding  what  information  they 
consider  important  for  decision  making  purposes,  and  how  they  want  that  information 
presented  are  proliferating  rapidly.  The  underlying  assumption  is  that  consumers  will 
use  this  information  to  make  more  informed  and  appropriate  choices  among  the  health 
plans  available  to  them,  leading  to  improved  quality  and  cost  containment  (U.S.  General 
Accounting  Office,  1995;  Schnaier  et  al.,  1995).  Previous  research  indicates  that  while 
consumers  want  comparative  information  on  plan  characteristics,  including  data  on 
quality  of  care  (Agency  for  Health  Care  Policy  and  Research,  1995;  National  Committee 
on  Quality  Assurance,  1995),  they  may  conceptualize  quality  of  care  in  ways  that  differ 
markedly  from  traditional  approaches  used  by  differently  large  scale  purchasers  and  the 
research  community.  For  example,  consumers  are  more  likely  to  define  quality  of  care 
in  terms  of  the  interpersonal  relationship  with  their  provider  rather  than  the  clinical 
processes  and  outcomes  of  care  (Mechanic,  1989).  Furthermore,  consumers  may  fail 
to  understand  key  aspects  of  the  indicators  provided  (Hibbard  and  Jewett,  1996). 

The  primary  purposes  of  our  research  were  to:  (1 )  identify  consumer  information 
needs  and  preferences  for  choosing  and  using  health  plans  and  (2)  develop  and  test 
prototype  materials  providing  information  to  meet  these  empirically  determined  needs 
and  preferences.  We  collected  data  from  Medicaid  beneficiaries,  Medicare  benefici- 
aries, and  the  general  population  under  65  years  of  age.  However,  informational 
materials  were  developed  for  only  the  Medicaid  and  Medicare  populations.  Empirical 
data  collection  strategies  to  determine  consumers'  information  needs  and  preferences 
included  focus  groups  and  case  study  analysis.  Once  developed,  materials  were 
subjected  to  cognitive  testing  methodology  and  revised  based  on  the  test  results. 

1.2  Report  Format 

The  research  involved  four  distinct,  sequential  steps: 

•  Determine  the  information  needs  and  preferences  consumers  have 
for  choosing  among,  and  using,  health  plans. 

Develop  prototype  materials  targeting  those  information  needs. 

•  Empirically  test  the  prototype  materials  in  terms  of  their 
effectiveness  for  communicating  information  to  consumers  and 
enhancing  decision  making. 

Revise  the  prototype  materials  to  make  them  more  useful  and 
comprehensible  to  the  targeted  consumer  groups  based  on  the 
results  from  consumer  testing. 
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The  remainder  of  this  report  is  organized  according  to  these  sequential  steps, 
which  were  conducted  between  October  1 994  and  October  1 997.  We  begin  with  a 
summary  of  the  methodology  and  results  of  our  focus  group  and  case  study  research 
used  to  determine  consumer  preferences  and  information  needs  and  to  learn  about 
consumer  materials  being  developed  across  the  country.  We  then  briefly  describe  the 
decision  process  leading  to  the  materials  development.  Next  we  describe  our  cognitive 
testing  approacn  and  the  findings  from  this  research  as  well  as  how  the  materials  were 
modified  in  response.  We  conclude  with  a  presentation  of  the  final  set  of  materials 
which  are  appended. 
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Chapter  2:  Determining  Consumers'  Information  Needs  and 
Preferences 


2.1  Chapter  Overview 

Three  different  types  of  activities  were  used  to  assess  what  consumers  want  to 
know  and  how  they  prefer  to  receive  information:  (1)  a  literature  review,  (2)  focus 
groups,  and  (3)  case  studies.  A  comprehensive  literature  review  (Schnaier, 
McCormack,  Burrus  et  al.,  1994)  was  conducted  to  assess  the  availability  of  relevant 
written  documentation.  Because  of  the  ongoing  expansion  of  unpublished  information 
that  coincided  with  the  start  of  this  project,  we  used  case  studies  of  communities 
actively  engaged  in  the  use  of  report  cards  to  collect  additional  information.  We  also 
conducted  focus  groups  to  determine  preferences  from  the  consumer's  perspective.  In 
this  chapter  we  describe  the  methodology  and  results  of  our  focus  group  research 
followed  by  a  summary  of  our  case  study  analysis. 

2.2  Focus  Group  Methodology  and  Findings 

The  purpose  of  the  focus  groups  was  to  identify  the  factors  consumers 
considered  important  when  choosing  a  health  plan.  We  also  explored  several  factors 
that  limit  consumers'  acceptance  of,  understanding  of,  and  willingness  to  use 
performance  measures.1 

2.2.1  Overview  of  Focus  Group  Methodology 

Focus  groups  have  been  used  increasingly  in  social  science  research  as  a 
qualitative  method  of  gathering  information  on  people's  opinions  in  greater  detail  than  is 
typically  available  through  survey  techniques,  with  some  economies  of  both  time  and 
resources.  A  focus  group  discussion  is  led  by  a  trained  moderator  with  approximately  8 
to  12  persons  participating  (Burrus  et  al.,  1992).  The  group  interaction  within  the  focus 
group  is  designed  to  simulate  naturally  occurring  discussions  on  a  topic  with  the 
exchange  among  group  members  evoking  ideas  and  reactions  that  might  not  have 
been  considered  in  an  individual  interview.  Systematic  analysis  reveals  key  "themes" 
that  reflect  a  synthesis  of  the  opinions  expressed  across  groups  and  types  of  individuals 
for  the  topic  of  interest. 

Despite  all  the  advantages  of  focus  groups,  those  reviewing  the  results  should 
keep  certain  caveats  in  mind.  First,  as  with  other  data  collection  methods,  selection 
bias  is  a  threat  to  validity  if  there  is  any  systematic  bias  created  by  either  participant 
recruitment  strategies  or  the  type  of  individual  who  agrees  to  participate.  Even  if  focus 
group  participants  are  recruited  randomly,  the  extent  of  participation  bias  is  difficult  to 
assess  and  limits  researchers'  ability  to  extrapolate  or  generalize  results  to  broader 


1  For  more  detailed  information  on  the  focus  group  methodology  and  results,  please  see  Gibbs, 
Sangl,  and  Burrus,  1997  and  Gibbs,  1996. 
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populations.  The  group  format  means  that  participation  bias  may  also  be  a  threat  if 
discussions  are  dominated  by  persons  representing  particular  viewpoints.  Moreover, 
the  highly  interactive  setting  increases  the  possibility  of  bias  if  individuals  feel  inclined  to 
offer  what  are  perceived  to  be  socially  desirable  responses  or  if  there  is  a  generalized 
inhibition  about  discussing  certain  topics.  Furthermore,  because  focus  groups  are  so 
strongly  influenced  by  the  dynamics  of  the  group,  the  "group,"  rather  than  the 
individuals  within  the  group,  is  the  most  appropriate  unit  of  analysis.  The  typically  small 
study  population  (in  this  case,  22  groups)  means  that  focus  group  studies  are  generally 
inappropriate  for  identifying  differences  among  subpopulations  although  "trends"  may 
be  suggested.  Although  using  a  skilled  and  culturally  sensitive  moderator,  conducting 
multiple  groups,  as  well  as  using  systematic  analytic  strategies,  can  help  mitigate  these 
problems,  the  above  caveats  should  be  kept  in  mind  when  reviewing  the  analysis  and 
interpretation  of  focus  group  data. 

2.2.2  Focus  Group  Methodology 

Three  insurance  populations  (Medicare,  Medicaid,  and  privately  insured 
individuals  under  55  years  of  age)  were  included  in  the  study  for  a  total  of  22  focus 
groups  at  8  different  locations,  as  shown  in  Table  1 .  Within  each  insurance  population, 
we  recruited  white,  African-American,  and  Hispanic  participants,  as  well  as  persons  with 
and  without  chronic  illnesses.  To  increase  within-group  homogeneity  and  topic 
salience,  we  specified  that  participants  would  be  middle  income  (defined  as  within  one 
quartile  of  the  median  income  for  their  city  or,  for  rural  sites,  county).  Second,  we 
limited  participation  to  persons  who  had  had  the  opportunity  to  make  a  choice  among 
plans  either  by  new  enrollment  or  reenrollment  within  the  past  2  years.  For  privately 
insured  consumers,  this  meant  that  participants  must  have  had  more  than  one  health 
plan  to  choose  from  through  employment  or  individual  purchase.  For  Medicare 
beneficiaries,  we  included  only  persons  who  had  purchased  a  Medicare  supplement, 
enrolled  in  a  Medicare  Health  Maintenance  Organization  (HMO),  or  given  serious 
consideration  to  changing  plans  within  the  past  two  years.  For  Medicaid  enrollees, 
groups  were  held  in  States  where  enrollees  are  formally  offered  the  chance  to  change 
their  health  plan  annually  and  are  able  to  initiate  procedures  to  change  plans  at  any 
time.  Third,  we  increased  the  likelihood  of  insurance  coverage  being  a  highly  salient 
issue  within  our  privately  insured  groups  by  screening  to  ensure  that  participants  had  at 
least  one  child  living  at  home  and/or  had  a  family  member  with  a  chronic  physical 
condition,  thereby  increasing  the  likelihood  of  encounters  with  the  health  care  system 
and  familiarity  of  health  insurance  issues.  Fourth,  all  Medicare  beneficiary  and  privately 
insured  focus  groups  were  held  in  sites  with  moderate  managed  care  penetration,  so 
that  participants  would  be  able  to  discuss  their  own  considerations  and  experiences  in 
assessing  managed  care  plans  and  deciding  among  managed  care  and  fee-for-service 
(FFS)  options.  Both  States  had  mature  managed  care  plans  programs  where  startup 
issues  had  already  been  addressed.  Finally,  our  group  configuration,  site  selection, 
and  recruiting/screening  strategies  were  planned  to  include  residents  of  major 
metropolitan  areas,  smaller  cities  in  different  geographic  regions,  racial  and  ethnic 
minorities.  Each  type  of  population  was  interviewed  in  at  least  two  different  geographic 
sites  (except  in  two  instances)  to  minimize  potential  regional  bias.  We  attempted  to 
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represent  new  Medicare  beneficiaries  by  including  two  groups  of  persons  63  or  64 
years  of  age  at  the  time  the  groups  met. 

The  moderator  guides  were  similar  for  each  population  to  maximize  compara- 
bility across  groups,  although  the  guides  were  tailored  to  take  into  consideration  the 
unique  features  of  each  population's  insurance  situation.  Moderators  with  substantive 
expertise  in  health  care  issues  as  well  as  extensive  experience  in  focus  group  methods 
led  each  of  the  groups.  To  assist  the  moderator,  a  second  team  member — also  with 
both  substantive  and  methodological  expertise — served  as  the  notetaker  and  logistic 
support  person  for  each  group.  With  participants'  permisison,  all  groups  were  audio 
taped.  We  did  not  use  videotape  to  minimize  reluctance  on  the  part  of  participants  to 
speak  openly. 

The  protocol  focused  on  the  following  issues: 

Dimensions  of  the  health  plans  most  important  to  consumers  in 
choosing  a  plan 

•  Decision  processes  and  stages  for  plan  selection  and  information 
required 

Comparative  information  used  for  making  choices,  examination  of 
sample  presentations  of  consumer  satisfaction  ratings  and  clinical 
performance  measures,  and  discussions  of  whether  these  would 
be  useful  for  plan  selection 

•  Credible  information  sources 

•  Problems  encountered  in  using  health  plans  and  the  kinds  of 
information  needed  to  resolve  them. 

Following  transcription  of  all  audio-taped  groups,  project  staff  coded  text 
segments  of  content  area,  speaker  characteristics,  and  group  parameters.  A  text- 
oriented  database  software  package  (AskSam)  was  then  used  to  sort  coded  data 
segments  for  review,  allowing  us  to  examine  all  of  the  statements  on  a  given  topic  (e.g., 
physician  choice)  and  to  compare  statements  systematically  according  to  participant 
characteristics  (e.g.,  presence  or  absence  of  chronic  condition  in  the  household). 

Emerging  themes  were  identified  and  subjected  to  further  coding  and  review. 
Codes  were  reviewed  to  check  reliability  across  raters. 

2.2.3  Focus  Group  Findings 

Choice  Process  and  Information  Needs 

Most  participants  described  the  health  insurance  decision  making  process  as 
frustrating  and  difficult.  Medicare  beneficiaries  apparently  experienced  a  heightened 
awareness  of  the  seriousness  of  the  process  due  to  concerns  about  their  own  or  their 
contemporaries'  increasing  physical  limitations;  and  in  some  cases  they  had  concerns 
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about  their  own  decreasing  cognitive  abilities.  Others  spoke  of  feeling  poorly  equipped 
to  negotiate  the  health  plan  decision  process  formerly  handled  by  a  spouse  or 
employer.  Perhaps  because  of  this  increased  perception  of  the  vulnerability  in  their 
health  status,  we  did  not  detect  any  identifiable  differences  between  Medicare 
beneficiaries  with  chronic  disease  and  those  without. 

Concern  and  understanding  regarding  impending  coverage  decisions  was  very 
low  among  those  who  had  not  yet  reached  age  65  or  had  a  spouse  reach  that  age. 
Once  age  65  had  been  reached,  however,  Medicare  beneficiaries  demonstrated  a  keen 
interest  in  learning  more  about  their  choice  options,  although  many  expressed 
frustration  at  the  lack  of  information  available  to  help  guide  their  decision  making. 

Beyond  the  basic  health  plan  features  of  benefits,  premium  costs,  and  the 
amount  of  paperwork  required,  Medicare  beneficiaries  identified  several  other  plan 
characteristics  of  interest.  Access  to  specific  providers  (e.g.,  "Can  I  keep  my  current 
doctor")  was  mentioned  most  frequently.  Although  the  beneficiaries  described  provider 
choice  in  terms  of  the  ability  to  see  a  doctor  who  knew  their  health  history,  their 
comments  clearly  indicated  the  relationship  also  represented  an  established  trust  in  the 
quality  of  care  provided  and  an  interpersonal  bond.  Although  the  desire  to  continue 
seeing  the  same  primary  care  physician  was  the  main  concern  in  this  area, 
beneficiaries  also  mentioned  access  to  specific  hospitals  or  specialists. 

Convenience  of  location  was  the  single-most  frequently  mentioned  factor  by 
Medicaid  recipients  (most  likely  due  to  the  fact  many  depend  on  public  transportation 
for  both  routine  and  urgent  care).  While  recipients  had  strong  feelings  about  other 
aspects  of  care,  their  choices  often  were  constrained  by  transportation  needs.  Like 
Medicare  beneficiaries,  Medicaid  enrollees  also  cited  the  ability  to  see  the  doctor  of 
one's  choice  as  an  important  factor  in  the  decision  process,  especially  for  those  with 
chronic  diseases  in  the  family.  They  also  described  waiting  time  for  both  routine  and 
urgent  care  as  important  considerations. 

Compared  with  Medicare  beneficiaries  and  Medicaid  enrollees,  privately  insured 
consumers  typically  have  fewer  health  plan  choices  (Institute  of  Medicine,  1996). 
Privately  insured  participants  with  chronic  diseases  were  acutely  aware  of  both  their 
requirements  for  coverage  and  the  restrictions  placed  on  their  choices  by  employers. 

"It  would  be  beneficial  in  a  lot  of  ways  to  choose  another 
plan  to  meet  the  rest  of  the  family's  needs,  but  we  always 
have  to  make  sure  my  daughter  gets  what  she  needs.  And, 
so  I  really  feel  trapped  a  lot  of  times." 

In  addition,  privately  insured  consumers  with  chronic  diseases  in  their  family  were  more 
interested  in  the  details  of  benefits  and  coverage  than  were  any  other  group.  Although 
price  was  generally  more  salient  for  privately  insured  consumers,  choice  of  a  physician 
remained  a  key  concern,  even  for  those  without  chronic  diseases  to  consider.  Provider 
choice  for  participants  with  chronic  disease  was  defined  primarily  in  terms  of  access  to 
specialist  care. 
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Response  to  Consumer  Satisfaction  Ratings 

Two  samples  of  health  plan  report  cards  were  provided  to  each  focus  group, 
beginning  with  a  simple  presentation  that  showed  scores  for  four  competing  (ficticious) 
health  plans  in  six  major  categories:  communication,  after-hours/urgent  care,  choice  of 
doctors,  technical  quality,  waiting  times,  and  customer  service.  Across  all  groups,  and 
especially  in  the  Medicaid  group,  we  found  that  some  participants  needed  explanations 
about  how  the  consumer  survey  (theoretically  used  to  collect  the  performance  data 
information)  was  conducted.  Some  participants  wanted  detailed  information  about  the 
survey  administration,  such  as  how  many  people  completed  the  survey,  who  the 
respondents  were,  and  whether  the  survey  was  performed  by  a  neutral  party. 

"Who  were  these  people — 5  people  employed  by  the 
insurance  company?" 

Typically,  participants  generally  focused  on  the  performance  measures  most  relevant  to 
their  personal  or  family  priorities — often  focusing  on  these  measures  to  the  exclusion  of 
the  others.  In  several  groups  (especially  among  Medicaid  enrollees),  participants 
spontaneously  reported  they  would  prefer  to  hear  the  opinions  of  an  individual  plan 
member  rather  than  examining  aggregated  consumer  data.  In  response  to  probing 
from  the  moderator,  they  agreed  that  any  particular  person's  responses  might  present  a 
biased  perspective  and  proposed  instead  that  several  people  be  assembled  to  relate 
their  experiences,  either  as  a  panel  or  video  presentation.  While  acknowledging  that 
such  spokespersons  might  not  be  sincere,  they  felt  more  confident  in  their  ability  to 
assess  the  truthfulness  of  individuals  in  a  face-to-face  situation  rather  than  to  evaluate 
numerical  ratings  on  paper. 

Response  to  Clinical  Performance  Measures 

We  also  asked  participants  to  review  a  sample  presentation  of  clinical 
performance  (e.g.,  HEDIS)  measures.  Reactions  to  this  piece  were  sharply  divided. 
Across  all  insurance  groups,  the  majority  of  participants  considered  compliance  with 
recommended  preventive  care  measures  to  be  the  consumer's  responsibility  and 
therefore  not  reflective  of  the  quality  of  care  delivered  by  the  plan.  Comments  such  as 

"If  you're  a  good  parent,  you're  going  to  remember  to  get 
your  kids  shots." 

were  typical.  Medicare  beneficiaries,  in  particular,  seemed  to  assume  that  periodic 
screenings  would  be  offered  by  their  physician  or  that  they  would  receive  them  if 
requested.  They  did  not  see  service  monitoring  as  a  plan  responsibility. 

For  ratings  of  both  technical  quality  of  care  and  consumer  satisfaction,  some 
participants  expressed  concerns  over  interpretation  of  the  numerical  ratings.  Although 
the  performance  measure  scores  were  carefully  selected  to  reflect  realistic  plan 
differences,  participants  questioned  both  the  generally  high  satisfaction  scores 
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"If  they're  all  around  80%  or  90%,  I  probably  wouldn't  pay  a 
whole  lot  of  attention  to  it." 

as  well  as  to  small  variations  among  plans. 

"When  it  comes  to  your  health,  even  small  differences  are 
important." 

Some  participants  did  not  infer  the  expected  causal  connections  between 
activities  of  the  health  plan  and  members'  health  outcomes.  Variations  among  plans  in 
rates  of  primary  care  utilization  or  low  birth  weight  babies,  for  example,  were  interpreted 
as  differences  in  the  population  covered  by  the  plan,  rather  than  as  plan  variation.  In 
addition,  some  performance  measures  intended  to  be  positive  were  interpreted  as 
negative.  For  example,  Medicare  beneficiaries  expressed  concerns  that  a  plan  with 
lower  hospitalizations  for  pneumonia  was  underrating  its  patients  who  should  have 
been  hospitalized.  A  Medicaid  enrollee  believed  that  a  low  ceasarean  section  rate 
meant  that  the  plan  was  denying  women  the  aggressive  care  they  needed.  The 
respondents  perceived  low  rates  as  signals  that  they  might  encounter  barriers  to 
needed  care.  Given  the  financial  incentive  health  plans  have  to  discourage  utilization, 
this  finding  suggests  that  utilization-based  measures  of  performance  in  which  lower 
rates  are  considered  to  reflect  higher  quality  should  be  excluded  from  report  cards.  If 
they  are  included,  they  should  be  used  together  with  satisfaction  measures. 

We  found  that  a  general  lack  of  understanding  of  how  a  health  plan  might  influ- 
ence member  behavior  to  be  a  barrier  to  effective  communication.  Medicaid  enrollees 
strongly  believed  individuals  should  be  responsible  for  complying  with  recommended 
screenings  and  parents  should  be  responsible  for  keeping  their  children's  care  up  to 
date.  Medicare  beneficiaries  typically  expressed  bewilderment  that  anyone  would  fail  to 
take  advantage  of  covered  preventive  services.  Many  Medicare  beneficiaries  and 
Medicaid  enrollees  also  did  not  recognize  a  plan's  potential  for  modifying  physician 
behavior.  Responsibility  for  the  process  of  care  was  almost  exclusively  attributed  to  the 
individual  physician  although  negative  experiences  were  frequently  seen  as  reflective  of 
the  plan,  particularly  if  more  than  one  physician  was  involved.  Similarly,  difficulties  in 
accessing  care  were  likely  to'  be  blamed  on  the  plan,  while  ready  access  was  often 
attributed  to  the  physician's  willingness  to  extend  special  considerations  on  behalf  of 
the  patient.  Many  Medicaid  enrollees  as  well  as  some  Medicare  beneficiaries  typically 
selected  and  rejected  performance  measures  based  on  their  specific  needs,  rather  than 
inferring  a  more  generalized  pattern  of  care  such  indicators  are  meant  to  represent. 

Response  to  Cost  Profiles 

To  explore  the  potential  usefulness  of  expanding  cost  information  beyond 
standard  comparisons  of  premiums,  deductibles,  and  copayments,  we  asked 
participants  whether  presentation  of  average  costs  per  year  for  "typical"  patients,  or  for 
patients  with  various  chronic  conditions,  would  be  of  interest.  Participants  quickly 
pointed  out  that  a  typical  consumer  was  unlikely  to  exist  although  a  few  were  interested 
in  the  idea  of  seeing  how  costs  would  compare  under  different  plans.  Participants  with 
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chronic  diseases  were  least  likely  to  be  interested  in  this  sort  of  example.  They  were 
more  likely  to  have  examined  their  plan's  structure  and  their  own  projections  for  their 
future  costs  in  detail  and  did  not  expect  that  any  example  could  represent  the  unique 
features  of  their  particular  circumstances.  Some  Medicare  beneficiaries  had  developed 
elaborate  worksheets  with  which  to  itemize  anticipated  utilization  of  various  services 
and  to  compare  resulting  costs  under  the  plans  available  to  them. 

Information  Sources 

All  types  of  participants  overwhelmingly  preferred  impartial  information  sources, 
although  variations  existed  among  participants'  concepts  of  who  would  be  the  most 
trustworthy  and  informed  source.  Information  from  relatives  and  friends  was  seen  as 
highly  credible  and  useful  and  often  was  preferred  over  published  information. 
Opinions  were  mixed  on  the  credibility  of  government  agencies,  particularly  among 
minority  participants.  Programs  sponsored  by  state  and  Federal  government,  such  as 
the  Medicare  Information,  Counseling,  and  Assistance  Programs  (ICAs),  were 
spontaneously  named  as  trustworthy  sources,  but  the  Federal  government  was  viewed 
with  caution. 

Most  felt  that  insurance  plan  representatives  were  not  .trustworthy  resources  and 
expressed  skepticism  of  information  prepared  by  insurers. 

"If  you  have  somebody  from  the  plan,  they're  just  lining  their 
own  wallets" 

"I  think  ...they  pad  these  things  to  make  themselves  look 
good." 

This  distrust  extended  to  the  American  Association  of  Retired  Persons  (AARP),  which 
was  suspect  as  an  information  resource  because  of  the  organization's  involvement  in 
marketing  Medicare  supplemental  insurance.  The  magazine,  Consumer  Reports, 
published  by  Consumers  Union,  was  frequently  cited  as  a  credible  and  useful  resource. 

Problems  in  Using  Health  Plans 

Difficulties  in  using  health  plans  were  articulated  by  many  of  the  participants. 
Problems  cited  most  often  included  denial  of  payment  for  emergency  room  care  and 
long  waits  for  service  (Medicaid  enrollees)  and  limited  access  to  specialists  (Medicare 
beneficiaries  and  chronically  ill  participants).  Although  many  had  been  successful  in 
resolving  their  difficulties,  they  also  expressed  feelings  of  powerlessness,  particularly 
for  recurrent  problems. 

"I  have  to  fight  for  every  single  thing." 

Participants  tended  to  view  these  difficulties  as  isolated  problematic  interactions  with  a 
provider  or  plan.  Few  expressed  any  concept  of  consumer's  rights  or  knowledge  of 
established  procedures  for  problem  resolution. 
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2.2.4  Focus  Group  Conclusions 
Information  Needs 

Our  focus  groups  clearly  indicated  a  desire  on  the  part  of  consumers  across  ail 
insurance  groups  for  more  information  to  assist  them  with  plan  choice.  There  were 
general  patterns  within  groups  as  to  particular  areas  of  interest.  Medicare  beneficiaries 
wanted  to  know  whether  they  would  have  continued  access  to  their  current  provider  and 
to  specialist  care,  and  how  much  protection  from  financial  risk  the  plan  offers.  Pre- 
Medicare  beneficiaries  were  eager  for  basic  information  to  help  guide  their  decisions 
regarding  ways  to  supplement  Medicare  coverage.  Medicaid  enrollees  wanted  to  make 
certain  they  would  be  able  to  access  needed  care  and  assurances  they  would  be 
treated  with  respect. 

Our  findings  suggested  that  some  current  beneficiaries,  and  many  approaching 
Medicare  eligibility,  are  confused  by  the  complex  financial  structure  of  the  available 
coverage  options.  Given  the  complexity  of  Medicare  payment  system  in  combination 
with  various  fee-for-serv'ce  and  managed  care  plan  options,  presentations  that  clearly 
outline  beneficiaries'  choices  and  explain  the  benefits  and  limitations  of  each  would  be 
useful  for  addressing  this  perceived  information  shortfall. 

Performance  Measures 

Although  participants  indicated  only  a  limited  interest  in  performance  measures, 
we  believe  much  of  this  hesitation  stemmed  from  a  lack  of  familiarity  and  appreciation 
of  what  these  measures  offer  and  their  appropriate  usage.  Basic  explanations  of  what 
these  measures  represent  and  how  they  are  compiled  would  be  necessary  if 
consumers  are  to  understand  and  use  them.  The  relationship  between  performance 
measures  and  the  processes  and  outcomes  of  care  needs  to  be  made  explicit. 
Although  use  of  nontechnical  terminology  is  a  good  starting  point,  most  audiences  will 
likely  require  explanations  of  why  a  measure  represents  a  desirable  or  undesirable 
event  and  examples  of  how  to  interpret  relative  scores.  The  interest  of  Medicaid 
enrollees  in  hearing  the  experience  of  individuals  rather  than  relying  on  data  suggests 
that  first-person  quotes,  in-person  presentations,  or  use  of  videos  with  "peer"  actors 
could  be  effective  for  framing  performance  measure  presentations  for  consumers  who 
are  less  quantitatively  oriented.  The  information  should  also  assist  users  in  interpreting 
the  relationship  between  the  performance  measures  and  the  health  plans  policies  and 
practices.  Without  examples  to  clarify  the  plan's  role  in  shaping  care  delivery, 
consumers  may  attribute  events  exclusively  to  physicians  and  patients. 

The  measures  presented  should  be  customized  to  the  health  priorities  of  the 
insurance  group.  For  example,  seniors  saw  indicators  such  as  mammography  rates 
and  cholesterol  screenings  as  being  particularly  relevant  to  their  care,  but  families  with 
young  children  were  more  interested  in  the  use  of  prenatal  care  and  immunizations. 

Descriptions  of  methodology  should  demonstrate  that  the  data  are  collected 
impartially  and  that  the  results  are  valid.  Some,  although  not  all,  consumers  will  want 
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the  details  of  consumer  survey  administration,  such  as  sample  size  and  selection 
methods,  response  rate,  who  administered  the  instrument,  and  who  analyzed  or 
audited  the  results.  Consumers  are  unlikely  to  accept  or  use  information  they  perceive 
to  be  biased.  In  addition,  consumers  want  assurances  that  the  data  collection  process 
was  fair  and  uniform.  Without  evidence  of  standardization,  they  are  likely  to  interpret 
the  data  as  including  only  the  positive  results  or  information  that  places  the  plans  in  a 
favorable  light. 

Consumers'  difficulties  in  understanding  the  values  associated  with  ratings  is 
also  a  barrier  that  must  be  overcome.  The  focus  groups  suggested  that,  even  if  issues 
of  statistical  significance  are  not  directly  addressed,  consumers  make  their  own 
assessments  of  how  much  variation  in  ratings  across  plans  is  meaningful.  Participants 
were  most  enthusiastic  about  performance  measures  that  allowed  them  to  identify 
plans  that  were  clearly  outstanding  or  inferior.  While  statistical  significance  of 
differences  among  plans  is  important  for  policy  reasons,  it  may  be  difficult  for 
consumers  to  understand  because,  with  large  enough  samples,  small  differences  may 
be  statistically  significant  but  may  not  have  practical  significance  for  consumers. 

Media  Preferences 

Responses  showed  a  clear  pattern  of  media  preferences.  For  Medicare 
beneficiaries,  the  overwhelming  choice  was  a  group  presentation,  supplemented  by 
printed  materials.  Our  findings  showed  little  inclination  toward  the  use  of  new 
information  technologies  among  current  Medicare  beneficiaries,  although  these  will 
likely  become  more  attractive  as  people  more  experienced  with  computers  and  video 
technologies  become  eligible  for  Medicare.  Further,  the  Medicare  beneficiaries  also 
indicated  a  clear  preference  for  having  written  materials  that  they  can  use  on  their  own, 
refer  back  to,  and  work  through  at  their  own  pace  as  needed. 

Medicaid  enrollees  indicated  general  dissatisfaction  with  group  presentations 
that  were  in  use  at  the  time  of  the  focus  groups,  although  this  may  be  more  driven  by  a 
lack  of  substantive  information  to  guide  presentation  than  by  the  presentation  media. 
Enrollees  indicated  a  preference  for  receiving  written  materials  to  review  privately  and 
follow-up  with  questions  before  making  a  choice.  Most  indicated  they  would  make  use 
of  a  telephone  hotline  as  an  adjunct  to  the  written  or  personal  presentations. 

Participants  consistently  preferred  receiving  information  from  an  impartial  source. 
The  most  trustworthy  sources  appear  to  be  a  state  government  agency,  independent 
research  firm,  or  consumer  advocacy  group.  Medicaid  enrollees  repeatedly  expressed 
an  interest  in  hearing  information  from  persons  like  themselves.  Using  enrollees  as 
spokespersons  in  a  video  or  as  counselors  was  a  suggested  approach  to  increasing 
credibility. 

Since  consumers'  interest  in  documentation  varies  even  within  insurance  groups, 
information  would  ideally  be  presented  in  a  "layered"  fashion,  structured  to  allow  users 
to  choose  their  desired  level  of  simplicity  or  detail.  Basic  definitions  could  be  presented 
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in  the  body  of  the  report  with  more  detailed  explanations  available  for  those  seeking 
more  detailed  explanations. 

2.3    Case  Study  Research 

2.3.1  Case  Study  Methodology 

The  case  study  component  of  our  research  was  designed  to  gather  data  about 
and  from  organizations  throughout  the  country  that  were  developing  informational 
materials  to  assist  consumers  in  choosing  a  health  plan  and  in  using  the  health  care 
system.  It  provided  an  opportunity  to  identify  the  methods  organizations  use  to  transmit 
information  to  health  care  consumers,  the  process  by  which  they  determine  consumer 
information  needs,  and  the  effectiveness  of  their  approaches.  In-person  interviews 
were  conducted  with  employers,  health  maintenance  organizations  (HMOs),  health  care 
purchasing  cooperatives  and  coalitions,  national  magazines,  insurance  counseling 
programs,  an  accreditation  body,  a  hospital,  one  federal  agency,  and  several  State 
agencies.  Case  study  participants  were  selected  from  communities  with  active 
managed  care  markets,  in  which  several  organizations  were  providing  consumer  choice 
materials  to  at  least  one  of  our  three  target  populations.  The  five  case  study 
communities  were: 

•  Sacramento  and  San  Francisco/Oakland,  California; 
Minneapolis/St.  Paul,  Minnesota; 

•  Cleveland,  Ohio; 

New  York  City,  New  York/Stamford,  Connecticut;  and 

•  Washington,  DC. 

A  total  of  24  organizations  were  interviewed. 

Through  the  case  studies,  we  identified  a  list  of  candidate  performance 
measures  for  inclusion  in  our  prototype  materials  and  a  list  of  potential  formats/modes 
of  communication  for  materials  design.  To  evaluate  the  different  types  of  performance 
measures  found  in  the  numerous  materials  we  encountered,  measures  were  classified 
into  three  broad  categories:  (1)  structural  measures,  (2)  sua-ey-based  measures,  and 
(3)  administrative  records-based  measures.  Structural  measures  included  those  that 
characterize  basic  health  plan  features  such  as  the  benefits,  premiums,  provider 
networks,  and  operations  of  health  plans.  Survey-based  measures  reflect  feedback 
from  consumers  who  are  already  enrolled  in  the  plan,  their  experiences  and  level  of 
satisfaction.  The  survey-based  measures  were  organized  in  six  categories: 

•  Access  to  Care, 
Communication/Interpersonal  Skills, 

•  Experience  with  the  Physician/Hospital, 
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•  Quality  of  Care, 

•  Preventive  Services,  and 
Experience  with  the  Plan. 

Our  third  category,  records-based  measures,  are  derived  from  administrative  data 
bases  such  as  automated  medical  records,  claims,  encounter  data,  and  accounting 
data  about  services  delivered  by  the  health  plan.  We  categorized  administrative 
performance  measures  into  four  groups: 

Screening/Preventive  Care, 
Utilization, 

•  Outcomes,  and 
Administrative/Financial  Management. 

2.3.2  Case  Study  Findings 

In  reviewing  the  case  study  findings,  the  reader  should  keep  in  mind  that  the 
case  studies  were  conducted  in  the  spring  of  1 995  and  represent  a  historical 
documentation  of  the  state-of-the-art  at  that  point  in  time.  Substantial  progress  has 
been  made  in  the  materials  development  arena  since  that  time.  With  this  caveat  in 
mind,  we  present  the  general  findings  from  our  case  study  data  collection  efforts  which 
guided  our  materials  development  process.2 

Overall,  structural  data  elements  and  in  particular,  premium  amounts  and 
benefits  coverage,  were  the  most  common  type  of  information  included  in  the  consumer 
materials  that  we  reviewed.  This  is  also  the  most  basic  type  of  information,  distribution 
of  which  is  essentially  universal  among  health  plans  and  purchasing  organizations. 
Survey-based  measures  differ  from  structural  and  administrative  records-based 
measures  in  that  they  are  based  on  subjective  impressions  of  individuals'  health  care 
experiences  and  expectations  and  may  reflect  the  "values"  that  individuals  place  on  the 
components  of  the  health  care  experience.  Among  the  24  organizations  that  we 
interviewed,  about  one-half  had  completed  development  of  materials  that  included 
performance  measures  from  member  surveys.  Of  organizations  that  used  survey- 
based  measures,  a  majority  reported  six  or  fewer  measures.  Performance  reports  that 
were  designed  for  corporate  purchasers  generally  used  more  measures  than  those 
designed  especially  for  individual  consumers.  The  most  commonly  used  survey-based 
measures  were: 


For  more  detailed  information  on  the  case  studies,  see  McCormack,  Schnaier,  Garfinkel  et  al., 
1996.  "Consumer  Information  Development  &  Use."  (Fall,  1996).  Health  Care  Financing  Review,  1996, 
Vol.  18,  No.  1. 
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•  overall  satisfaction  with  health  care 

•  overall  satisfaction  with  the  insurance  plan  or  HMO 

•  satisfaction  with  the  range  or  services  covered 

•  overall  satisfaction  with  the  doctor  or  health  care  provider. 

In  several  cases,  these  measures  were  composites  of  several  other  measures, 
collapsed  to  create  an  overall  rating  to  decrease  the  total  number  of  measures  and 
enhance  consumer  usage. 

About  half  of  the  organizations  developed  consumer  materials  containing 
administrative  records-based  performance  measures.  Measures  reflecting  preventive 
health  screenings  were  the  most  common  type  used.  For  example,  nine  organizations 
presented  data  on  the  percent  of  women  who  received  a  mammogram  and  a  Pap 
smear  (at  the  appropriate  intervals).  Health  Plan  Employer  Data  Information  Set 
(HEDIS)  measures  were  prevalent  among  both  the  screening  and  utilization  measures 
found  in  the  materials  we  reviewed.  Seven  organizations  used  at  least  one  outcome 
measure;  low  birth  weight  rates  were  the  most  common  outcome  measure  found  in  the 
materials. 

The  case  studies  were  also  designed  to  identify  the  most  common  media  and 
layout  of  materials  used  to  present  health  care  choice  information  to  consumers.  The 
printed  "report  card"  was  the  most  common  format  encountered;  most  included  a 
combination  of  text  and  graphics.  Graphics  took  the  form  of  bar  charts,  icon  figures, 
and  circles  with  various  proportions  shaded  but  the  dominant  model  used  text  with 
percentages  reflecting  rating  scores  in  tabular  format.  Most  organizations  agreed  with 
the  need  for  simple,  straightforward  information  presented  in  both  text  and  pictorial 
format.  They  also  recognized  that  no  single  approach  was  appropriate  for  all 
audiences.  Only  one  organization  used  interactive  computer  software  to  provide 
information.  Live  group  presentations  were  used  with  the  Medicaid  population  in 
California.  Although  a  few  organizations  used  focus  group  findings  to  guide 
development  of  their  materials,  most  simply  relied  upon  their  professional  judgment. 
Material  format  was  largely  still  under  development  at  the  time  of  most  site  visits. 

We  also  explored  the  extent  to  which  the  materials  developers  (1)  conducted 
scientific  assessments  of  consumer  information  needs  and  (2)  evaluated  their 
materials.  We  found  little  systematic  investigation  of  either  issue.  A  majority  of  the 
organizations  that  we  interviewed  based  their  perceptions  of  consumer  information 
needs  for  choosing  a  health  plan  on  personal  and  organizational  experience.  When 
research  was  conducted  to  determine  consumer  needs,  it  was  primarily  through  focus 
groups.  Further,  materials  developed  by  the  case  study  organizations  had  not 
undergone  rigorous  evaluations;  most  had  not  been  evaluated  at  all.  Several 
organizations  were  planning  to  evaluate  their  materials,  with  focus  groups  being  the 
primary  method  for  these  anticipated  evaluations. 


14 


We  learned  several  lessons  by  conducting  the  case  studies  that  were  important 
for  guiding  the  development  of  the  prototype  materials.  First,  most  beneficiaries  were 
fairly  unfamiliar  with  health  plan  performance  measures  and  their  meanings,  suggesting 
that  performance  reports  should  be  kept  simple  in  the  initial  stages  of  use. 
Beneficiaries'  limited  experience  with  performance  reports  also  implies  that  the 
presentation  of  data  must  be  accompanied  by  simple  explanations  of  the  meaning, 
interpretation,  and  importance  of  the  data. 

The  level  of  detail  beneficiaries  can  understand  and  make  use  of  varies 
considerably  within  any  population.  Thus,  "layering"  information — which  involves 
presenting  the  same  information  in  two  or  three  different  ways,  each  with  progressively 
more  detail — may  be  most  appropriate  in  determining  how  much  information  to  provide. 
In  comparing  the  use  of  performance  measures  across  organizations,  we  discovered  no 
consistent  pattern  that  clearly  pointed  to  a  set  of  measures  for  inclusion  in  the 
materials.  Not  only  do  the  reported  measures  themselves  vary,  but  composite 
measures  that  appear  similar  in  name  often  represent  different  underlying  dimensions. 
We  decided  that  it  would  be  best  to  include  all  three  types  of  measures:  structural, 
survey-based,  and  statistical  measures  derived  from  the  administrative  databases  in 
our  prototype  materials  were  to  reflect  the  extant  materials  at  the  case  study  sites.  Our 
case  study  also  suggested  printed  materials  were  a  logical  starting  point  for 
communicating  choice  information  although  other  media  warranted  exploration.  We 
also  confirmed  the  importance  of  cognitive  testing  for  the  materials  we  developed, 
regardless  of  content  or  format.  Through  cognitive  testing,  consumers  would  define  for 
themselves  how  well  they  are  able  to  understand  and  use  the  materials.  Our  case  study 
results  suggested  materials  tested  through  rigorous  empirical  research  would  present  a 
novel  and  innovative  contribution  to  the  knowledge  of  consumer  usage  of  materials. 
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Chapter  3:  Materials  Development 


3.1    Decisions  About  the  Kinds  of  Prototypes  for  Development 

The  next  stage  of  the  project  was  to  develop  recommendations  for  the  kinds  of 
prototype  materials  to  be  developed.  In  making  our  decisions,  we  carefully  reviewed 
the  focus  group  and  case  study  findings  in  terms  of  implications  for  materials 
development.  We  also  presented  our  results  to  our  Technical  Advisory  Panel  (listed  in 
the  Acknowledgments  Section)  to  obtain  their  recommendations  for  prototype 
development  in  light  of  our  findings.  We  considered  other  factors  such  as  cost 
implications,  replicability,  and  feasibility  in  determining  the  media  format,  the  target 
audience,  and  the  type  of  information  to  present.  After  careful  consideration  of  this 
extensive  input,  we  selected  three  types  of  prototype  materials  to  develop: 

•       Videos  describing  health  plan  performance  measures  and  how  they 
can  be  used  by  Medicare  beneficiaries  and  Medicaid  enrollees 
when  choosing  a  health  plan. 

Handbooks  designed  to  explain  HMOs  to  Medicare  beneficiaries 
and  Medicaid  enrollees. 

A  handbook  with  worksheets  that  includes  a  step-by-step  process 
designed  to  help  persons  about  to  become  age  65  explore  options 
to  fill  in  the  gaps  in  Medicare  and  select  the  health  plan  best  suited 
to  their  needs. 

3.1 .1  Medicare  and  Medicaid  Videos  on  Performance  Measures 

Because  consumers  indicated  a  preference  for  "someone  like  me"  to  present  the 
information,  we  decided  a  video  would  be  the  best  format  for  addressing  this 
preference.  The  video  format  is  less  expensive  and  easier  to  use  logistically  than  a 
group  presentation,  yet  through  it,  the  feeling  of  having  one's  peers  discussing  the 
issues  can  be  captured.  Because  of  the  importance  of  tailoring  the  information  (i.e., 
using  performance  measures  that  are  relevant)  as  well  as  including  people  "like"  the 
viewer,  we  determined  separate  videos  would  be  needed  for  the  Medicare  and 
Medicaid  populations. 

Consumers  clearly  indicated  a  desire  for  more  information  to  help  inform  their 
decision  processes,  but  showed  only  limited  interest  in  performance  measures.  Thus, 
we  decided  to  draw  an  analogy  between  performance  measures  and  every  day 
purchase  decisions.  By  doing  this,  we  incorporated  the  concept  of  trade-offs  between 
quality  and  other  factors  such  as  cost.  This  helped  to  ground  the  decision  in  terms  that 
were  familiar  to  the  consumers,  helping  them  take  the  cognitive  step  from  familiar 
purchases  in  which  they  consider  quality,  to  choices  about  health  plans,  where  choice 
based  on  quality  was  less  familiar. 
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Finding  an  appropriate  comparative  purchase  decision  for  the  Medicaid 
population  was  difficult,  however,  in  that  the  major  purchases  (e.g.,  automobiles)  by  this 
population  are  typically  more  constrained  by  limited  financial  resources  than  for  non- 
Medicaid  populations.  After  much  deliberation,  we  decided  to  use  the  scenario  of  a 
grocery  shopping  experience,  in  which  consumers  are  forced  to  make  trade-offs 
decisions  between  costs,  quality,  and  personal  preferences. 

Two  video  scripts  were  prepared  and  submitted  to  the  Technical  Advisory  Panel, 
as  well  as  numerous  reviewers  from  HCFA,  for  comment.  These  comments  were 
reviewed  and  included  as  deemed  appropriate  by  the  HCFA  Project  Officer  and  project 
team.  The  video  was  then  recorded  in  a  real  supermarket,  using  age  appropriate 
actors.3 

3.1.2  Medicare  and  Medicaid  HMO  Handbooks 

Given  the  apparent  lack  of  consumer  knowledge  about  managed  care,  the 
growing  use  of  managed  care  options  for  both  Medicare  and  Medicaid  populations,  and 
the  importance  placed  by  focus  group  respondents  on  publications  that  the y  could 
share  with  family  and  friends  and  refer  to  over  time,  we  decided  to  develop  prototype 
written  materials  to  better  explain  managed  care  and  its  appropriate  use.  Although 
there  are  a  number  of  structurally  different  managed  care  options  available  (e.g.,  health 
maintenance  organizations  or  HMOs,  Preferred  Provider  Organizations  or  PPOs,  and 
Point  of  Service  Plans  or  POS),  the  prototypes  focus  on  HMOs.  The  written  materials 
explain  performance  measures  in  more  depth  than  the  videos,  and  also  provide 
examples  of  how  to  use  them. 

Based  on  the  focus  group  and  case  study  findings,  we  believed  that  these 
materials  should  be  provided  in  written  format  to  allow  consumers  to  review  them  on  an 
as-needed  basis.  Names  and  telephone  numbers  of  references  are  provided  for 
persons  seeking  additional  information. 

Draft  versions  of  the  printed  handbooks  were  submitted  to  numerous  HCFA  staff 
and  our  Technical  Advisory  Panel  for  review  and  comment.  The  HCFA  Project  Officer 
and  the  research  team  reviewed  these  comments  and  recommendations  and 
incorporated  them  into  the  materials  as  deemed  appropriate.  In  attempting  to  include 
many  of  the  comments  received,  we  were  faced  throughout  with  the  need  to  make 
trade-offs.  One  of  the  most  common  and  difficult  trade-offs  we  faced  was  the  need  to 
keep  the  materials  "user-friendly,"  while  also  addressing  reviewers'  concerns  that  the 
materials  should  be  technically  correct  and  comprehensive,  which  typically  makes  the 
materials  more  complex. 


3  Copies  of  the  Medicare  and  Medicaid  Videos  may  be  attained  from  HCFA. 
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3.1.3  Medicare  Options  Materials  for  Pre-Medicare  Beneficiaries 

The  final  prototype  we  prepared  was  directed  mainly  at  persons  about  to  become 
eligible  for  Medicare,  although  it  is  also  applicable  for  Medicare  beneficiaries 
considering  changes  in  supplemental  insurance  coverage.  People  in  this  age  group 
(60-64)  were  eager  for  information.  In  our  focus  groups,  we  found  that  they  were 
particularly  uninformed  about  what  Medicare  covers,  how  it  works,  and  what  options  are 
available  to  supplement  Medicare.  In  exploring  their  preferences  for  information,  it 
became  apparent  that  they  were  not  only  interested  in  the  factual  information  about 
options,  but  also  wanted  information  that  could  help  them  make  direct  comparisons  and 
trade-offs.  Some  people,  for  example,  brought  to  the  focus  groups  carefully  hand- 
written comparison  charts  they  had  developed  in  their  own  attempts  to  make  plan 
comparisons.  Based  on  these  observations,  we  decided  that  a  prototype  framework  for 
systematically  comparing  different  supplemental  insurance  options  would  be  useful. 
Although  existing  resources  for  current  Medicare  beneficiaries  may  be  applicable  in  the 
pre-Medicare  population,  very  limited,  if  any,  information  is  targeted  specifically  at  this 
age  group.  Because  the  health  insurance  decisions  that  must  be  made  upon  turning 
age  65  are  so  different  xrom  those  of  younger  persons  who  are  not  covered  by 
Medicare,  we  felt  that  development  of  a  separate  handbook  for  those  age  60-64  was 
warranted.  The  handbook  provides  a  brief  overview  of  Medicare,  but  focuses  on 
outlining  the  different  public,  individually  purchased  private,  and  employer-sponsored 
group  supplemental  insurance  options. 

We  did  not  develop  a  companion  video  to  accompany  the  Pre-Medicare 
beneficiary  handbook.  Instead,  the  handbook,  called  Medicare  Options:  Information  for 
New  Medicare  Beneficiaries,  contains  interactive  worksheets  that  help  people  assess 
the  trade-offs  between  the  different  supplemental  options  (e.g,  Medigap  versus  a 
Medicare  HMO),  select  one  of  the  options,  and  select  a  specific  health  plan  within  that 
option  category  (e.g.,  AARP  Medigap  plan  F). 

It  is  expected  that  pre-  and  current  Medicare  beneficiaries  will  find  both  the 
Medicare  Options  handbook  and  the  Medicare  HMO  handbook  useful,  because  they 
intentionally  complement  each  other.  For  example,  the  Medicare  HMO  handbook 
provides  HMO  performance  data;  the  Medicare  Options  handbook  does  not,  but  it 
provides  a  worksheet  for  choosing  among  the  whole  range  of  options.  Thus,  if  an 
individual  decides  by  reviewing  the  Options  handbook  that  he  prefers  a  Medicare  HMO, 
he  can  consult  the  Medicare  HMO  handbook  for  plan-level  performance  data.  Both 
handbooks  refer  readers  to  HCFA's  published  materials  such  as  the  Guide  to  Health 
Insurance  for  People  with  Medicare  and  Your  Medicare  Handbook.  They  also  refer 
users  to  the  Medicare  Information,  Counseling,  and  Assistance  (ICA)  programs  for  free 
objective  personal  assistance  with  decision  making.  In  fact,  several  references  to  these 
counseling  services  were  added  throughout  the  Options  handbook  to  make  it  easier  for 
users  to  obtain  personal  advice  and  information.  Many  beneficiaries  will  require 
personalized,  face-to-face  or  telephone  assistance. 
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3.2  What  is  New  in  the  Materials  Developed? 

The  materials  were  designed  to  address  several  areas  of  unmet  need  expressed 
by  consumers.  Focus  group  data  indicated  consumers  are  not  well  educated  about, 
nor  very  interested  in,  performance  measurement.  We  believed  this  stemmed,  in  part, 
from  a  lack  of  available  information  on  how  to  use  these  data  for  making  plan  decisions. 
To  address  this  apparent  shortfall,  a  primary  objective  of  the  materials  was  to  educate 
consumers  on  how  to  make  more  appropriate  and  informed  use  of  performance  data. 
Both  clinical  and  satisfaction  ratings  were  used  in  addition  to  basic  structural 
information  common  to  most  existing  materials.  The  materials  were  designed  to 
educate  consumers  about  how  these  data  reflect  health  plan  performance.  For 
example,  they  include  explanations  about  what  the  measures  mean,  how  to  interpret 
"high"  versus  "low"  numbers,  and  how  to  interpret  large  versus  small  differences. 

The  materials  also  were  designed  to  inform  consumers  about  choosing  among 
HMOs.  Most  of  the  materials  we  reviewed  in  our  case  studies  focused  on  presenting 
the  benefits  and  costs  of  various  plans  but  failed  to  provide  general  guidelines  for  the 
appropriate  selection  and  effective  use  of  HMOs.  The  prototype  materials  we  designed 
are  innovative  in  that  they  provide  a  process  whereby  consumers  can  determine  which 
plan  is  most  appropriate  for  them. 

The  third,  and  possibly  most  innovative,  contribution  of  these  materials  is  the 
attempt  to  provide  a  systematic  algorithm  for  comparing  traditional  fee-for-service  and 
managed  Medicare  options  in  a  written  format.  Almost  all  persons  approaching  age  65 
face  a  different  set  of  health  insurance  decisions  than  ever  before.  Most  will  have  to 
compare  alternatives  and  weigh  them  against  needs,  preferences,  and  constraints.  Yet 
our  focus  groups  revealed  that  they  often  have  little  idea  about  what  comparisons  need 
to  be  made  and  how  to  go  about  making  them  in  an  organized  manner.  Our  case 
studies  and  review  of  existing  materials  found  no  published  protocol  or  algorithm  to 
assist  beneficiaries  in  organizing  their  comparisons.  This  finding  was  expected 
because  the  concepts  that  need  to  be  conveyed  are  so  complex  and  the  "optimal" 
approach  varies  by  individual  according  to  knowledge  and  cognitive  style.  Several 
efforts  are  underway  to  develop  computer  algorithms  that  can  take  individual  variation 
in  style  and  preferences  into  account  more  easily  than  written  material.  But  because 
written  materials  are  so  important  to  this  population,  the  handbook  for  persons  about  to 
become  eligible  for  Medicare  presented  an  excellent  opportunity  to  design  a  prototype 
written  algorithm. 

3.3  What  the  Materials  Were  Not  Intended  to  Do 

The  materials  we  developed  for  this  project  were  not  intended  to  replace  the 
Guide  to  Health  Insurance  for  People  with  Medicare,  Your  Medicare  Handbook,  specific 
plan  guides,  or  numerous  other  resources  developed  for  Medicare  beneficiaries  or 
Medicaid  enrollees.  Rather,  these  materials  are  intended  for  use  in  conjunction  with, 
and  as  a  complement,  these  resources.  Similarly,  the  materials  are  not  designed  to 
replace  the  Medicare  ICA  counselors  or  Medicaid  enrollment  counselors.  We  believe 
that,  ideally,  these  materials  could  be  used  by  the  counselors  working  with  clients,  and 
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as  reference  materials  to  send  home  with  the  clients  for  ongoing  review  as  needed.  In 
short,  these  materials  are  not  intended  to  stand  alone. 

The  materials  were  not  intended  to  apply  to  all  the  possible  users.  The  Medicare 
video  and  handbooks  were  developed  for  the  aged  Medicare  population  and  the 
Medicaid  video  and  handbook  was  developed  for  the  AFDC  (Aid  to  Families  in 
Dependent  Children)  Medicaid  population  facing  mandatory  HMO  enrollment.  The 
special  needs  of  selected  disabled  groups  were  generally  not  considered  in  developing 
these  materials.  Because  of  the  diversity  of  needs  among  different  special  populations, 
the  HCFA  Project  Officer,  with  concurrence  from  the  remainder  of  the  research  team, 
decided  that  addressing  the  specific  needs  of  all  special  needs  populations  was  beyond 
the  scope  of  this  particular  project.  We  recommend  that  further  work  be  conducted  to 
tailor  the  kinds  of  choice  issues  presented,  as  well  as  presentation  media  used,  to 
address  the  information  and  presentation  needs  of  the  disabled  and  other  special 
needs  groups  that  we  were  unable  to  address.  We  encourage  recruiting  members  from 
these  populations  for  additional  focus  groups  to  determine  the  most  appropriate 
information  media  and  approaches  and,  subsequently,  to  develop  appropriate 
prototypes  that  address  the  special  consumer  needs  and  preferences  of  these  groups. 

Our  materials  were  developed  for  geographic  regions  with  at  least  a  moderate 
level  of  managed  care  penetration.  Parts  of  the  materials  (e.g.,  the  educational 
materials  on  performance  measures)  would  not  be  helpful  to  individuals  with  only 
choices  among  fee-for-service  plans,  thus,  modifications  would  be  required.  More  basic 
explanatory  information  on  managed  care  may  be  required  as  managed  care  begins  to 
penetrate  into  new  regions. 

The  eight  performance  measures  used  in  each  of  the  handbooks  are  only  place 
holders  and  are  not  intended  to  represent  our  "recommended"  measures.  Very  early  in 
the  project,  the  HCFA  Project  Officer,  with  concurrence  from  the  research  team, 
decided  that  determining  the  precise  measures  to  include  was  beyond  the  scope  of  this 
project.  Other  projects,  such  as  the  Consumer  Assessments  of  Health  Plans  Study 
(CAHPS),  are  currently  addressing  more  specifically  the  measures  that  should  be 
presented  to  beneficiaries. 

Finally,  these  handbooks  are  prototypes.  Because  they  are  designed  to 
compare  specific  HMOs  and  Medigap  insurers,  individual  versions  must  be  developed 
for  each  locale  in  which  they  are  used.  Statements  and  data  in  these  prototypes  should 
be  tailored  as  closely  as  possible  to  the  specific  rules,  choices,  and  data  that  exist  in 
each  locale. 
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Chapter  4:  Testing  and  Adapting  the  Materials 


4.1    Cognitive  Testing  Methodology 

Once  drafts  of  each  set  of  the  materials  were  prepared  and  reviewed  by  our 
Technical  Advisory  Panel,  we  subjected  them  to  rigorous  cognitive  testing.  Cognitive 
testing  methodology  is  used  to  investigate  how  people  react  to,  interpret,  and  use 
prototype  materials.  The  methodology  entails  a  set  of  interview  techniques  for 
investigating  consumers'  thought  processes  as  they  gather  information,  explore  their 
options,  and  make  decisions.  Some  examples  of  the  kinds  of  thought  processes  that 
may  be  involved  in  using  information  materials  to  make  a  health  care  decision  include 
comprehension  and  interpretation,  memory  recall,  comparison,  evaluation,  and 
selection  or  choice.  The  goal  of  our  cognitive  testing  research  was  to  identify  ways  to 
revise  the  prototype  materials  to  make  them  easier  to  use  and  more  helpful  for 
consumers.  .  . 

Our  cognitive  testing  addressed  four  general  areas: 

•  Materials  content:  Do  the  materials  contain  information  that  is 
relevant  to  health  care  consumers?  Is  the  information  sufficient? 
Is  the  information  complete?  Do  the  materials  contain  information 
that  seems  extraneous  to  consumers? 

•  Comprehension:  Do  consumers  understand  the  information 
provided?  Do  they  understand  the  information  as  intended? 

•  Navigation:  How  easily  do  consumers  work  through  the 
materials?  Do  the  materials  fit  together  in  ways  that  make  sense  to 
consumers?  Do  the  materials  help  consumers  identify  the 
information  they  want?  Do  the  materials  help  consumers  interpret 
the  information  they  find? 

•  Decision  Processes:  Do  consumers  understand  the  health  care 
choice  task  that  faces  them?  Do  they  recognize  the  roles  of  risk, 
costs,  and  utilities? 

Within  each  of  these  four  general  areas,  we  identified  more  specific  cognitive 
research  issues  and  selected  cognitive  testing  methods  for  addressing  them.  Testing 
procedures  also  were  tailored  to  the  unique  aspects  of  the  materials  as  well  as  the 
target  populations.  We  developed  a  "think-aloud"  interview  protocol  for  exploring  issues 
related  to  material  content  and  decision  making.  In  the  "think-aloud"  protocol, 
respondents  reviewed  the  materials  and  described  what  they  thought  about  as  they 
used  the  materials  to  make  a  choice  and  evaluate  options.  We  also  had  respondents 
consider  different  vignettes  as  they  used  the  materials  to  make  choices  and  evaluate 
options  for  the  character  in  the  vignette.  Feature  rating  tasks  were  also  used  where 
consumers  rate  how  important  the  various  pieces  of  information  were  to  their  health 
care  decision  process.  An  "observation  and  debriefing"  protocol  also  was  developed  for 
examining  comprehension  and  navigation.  In  the  observation  series  of  interviews, 
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participants  were  given  a  pretest  of  their  knowledge  and  then  were  observed  as  they 
read  sections  of  the  materials.  Interviewers  coded  various  aspects  of  their  behavior 
and  then  gave  a  posttest  for  comprehension  after  they  completed  their  review.  For  half 
of  the  participants  we  used  the  observational  protocol  and  for  the  other  half  we  used  the 
think-aloud  format.  Both  interviews  included  a  conjoint  task4  in  addition  to  the  cognitive 
testing  tasks. 

The  first  round  of  cognitive  testing  was  conducted  in  Portland,  Oregon,  and  the 
Maryland  suburbs  around  Washington,  DC.  These  areas  provided  geographic  diversity, 
but,  more  importantly,  also  had  a  moderate  penetration  of  managed  care.  Testing 
interviews  were  conducted  with: 

•  44  Medicaid  enrollees; 

•  44  Medicare  beneficiaries;  and 

•  15  Pre-Medicare  beneficiaries  (ages  60-64). 

The  first  round  of  cognitive  testing  produced  several  major  recommendations  for 
changes  to  the  draft  materials  as  developed.  These  issues  are  summarized  in  detail  in 
the  first  cognitive  testing  report  (RTI,  1996).  Based  on  the  results  and  recommenda- 
tions from  the  first  round  of  cognitive  testing,  significant  revisions  and  stylistic 
modifications  were  made  to  the  draft  materials.  The  second  round  of  cognitive  testing 
(contained  in  Appendix  B),  which  focused  on  the  written  materials,  was  limited  to  fewer 
participants  with  the  exception  of  the  Medicare  Options  handbook.  The  second  round 
of  testing  included: 

9  Medicaid  enrollees; 

•  9  Medicare  beneficiaries;  and 

•  24  Pre-Medicare  beneficiaries. 

The  Medicare  Options  handbook  was  tested  with  the  greatest  number  of 
subjects  because  these  materials  underwent  the  most  significant  revisions  following  the 
first  round  of  testing. 

The  second  round  of  testing  was  conducted  in  the  Washington,  DC  metropolitan 
area  only.  As  in  the  first  round  of  testing,  the  revised  drafts  of  the  prototype  materials 
were  tested  in  terms  of  content,  comprehension,  navigation,  and  the  overall  decision 
processes  participants  were  pursuing  while  using  the  materials.  Our  results  also 
contained  an  analysis  of  picture  and  color  preferences  and  perceived  overall  utility. 

The  third  and  fourth  rounds  of  cognitive  testing  focused  exclusively  on  the 
Medicare  Options  handbook  as  it  continued  to  evolve.  The  third  round  included  12 
interviews  with  participants  recruited  from  the  Washington,  DC  metropolitan  area. 


4  For  more  information  on  this  procedure  and  the  results,  please  see  Wood  et  al.  (1996).  We 
have  not  included  results  from  the  conjoint  testing  because  of  the  limited  number  of  subjects  used.  We 
do  suggest  that  based  on  our  testing,  conjoint  analysis  has  methodological  potential  and  warrants  further 
study. 
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Participants  were  recruited  from  a  variety  of  sources  including  the  local  ICA  program, 
area  senior  housing  facilities,  and  referrals  from  previous  testing  rounds.  To  assess  the 
final  booklet  version,  the  fourth  round  of  testing  used  a  small  group  format,  which 
included  a  group  of  five  lower  educational  levels  (defined  as  high  school  education  or 
less)  participants  and  a  group  of  seven  with  a  higher  educational  level  (defined  as  at 
least  some  college)  participants.  The  lower  educational  level  was  held  in  the 
Raleigh/Durham,  North  Carolina  area  and  the  higher  educational  level  was  held  in 
Washington,  DC.  In  round  four,  participants  reviewed  the  handbook  and  completed  the 
worksheet,  then  participated  in  a  discussion  about  them.  Before  leaving,  participants 
completed  a  short  survey  about  the  handbook.  In  all  rounds  of  testing,  we  focused  on 
recruiting  a  mix  of  participants  with  respect  to  race,  gender,  education,  and  HMO 
experience.  In  total,  51  cognitive  testing  interviews  and  2  small  group  discussions  were 
used  to  evaluate  the  Medicare  Options  handbook. 


4.2    Cognitive  Testing  Results 

We  have  extracted  the  most  relevant  results  and  "lessons  learned"  from  all  four 
rounds  of  cognitive  testing  and  discuss  them  in  the  following  sections  according  to 
target  population  and  type  of  material.5  Appendix  A  contains  the  complete  cognitive 
testing  reports  for  Rounds  2-4  and  depicts  changes  in  the  materials  based  on  our 
empirical  testing  results. 

4.2.1  Medicaid  Materials 

4.2.1.1  Medicaid  HMO  Video 

Overall,  the  Medicaid  enrollees  liked  the  video  and  perceived  it  as  "user- 
friendly."6  Most  reported  that  the  videotape  was  easy  to  understand  and  watch. 
Several  participants  remarked  that  likening  grocery  shopping  to  shopping  for  health 
care  seemed  appropriate  to  them.  There  were  no  reports  of  difficulties  understanding 
terminology  used  in  the  videotape.  Some  participants  complained  that  the  performance 
measure  charts  were  difficult  to  read,  although  this  may  have  been  a  result  of  the  small 
screens  on  the  portable  VCR  players  used  in  the  field.  The  beneficiaries  seemed  to 
respond  favorably  to  "people  like  them"  discussing  the  issues. 

One  of  the  most  significant  drawbacks  to  videos  is  the  difficulties  encountered 
when  modifications  are  needed.  For  example,  through  our  cognitive  testing  activities 
for  other  materials,  we  learned  beneficiaries  were  confused  by  the  term  "quality 
indicators,"  so  we  made  the  decision  to  use  the  term  "performance  measures,"  which 


5We  have  not  included  cognitive  testing  results  for  Round  1  because  the  materials  changed 
dramatically  from  Round  1  testing  to  Round  2,  driven  largely  the  testing  results  from  Round  1 . 

6  Because  of  the  significant  amount  of  material  to  be  tested,  we  did  not  have  adequate  time  to 
test  the  video  as  extensively  as  we  would  have  liked.  We  felt  it  was  important  to  keep  the  testing  to  less 
than  two  hours  and  made  the  decision  in  the  testing  design  phase  to  place  the  first  priority  on  testing  the 
written  materials  as  these  were  the  most  complex. 
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tested  better.  Making  this  same  change  to  the  charts  shown  in  the  video  to  illustrate 
different  quality  measures  would  have  been  relatively  simple.  However,  making  these 
changes  to  the  script  would  have  required  reenacting  several  scenes.  We  were  unable 
to  make  these  changes  due  to  funding  constraints.  In  hindsight,  we  believe  it  would 
have  been  better  to  produce  the  video  after  the  handbook  had  undergone  at  least  one 
round  of  cognitive  testing. 

4.2.1.2  Medicaid  HMO  Handbook 

Lessons  Learned  from  the  First  Round  of  Cognitive  Testing 

Need  to  Provide  Educational  Context.  The  first  version  of  the  Medicaid  HMO 
handbook  lacked  sufficient  introductory  text.  Several  participants  failed  to  understand 
that  the  text  and  graphical  comparison  charts  reflected  four  health  plans  from  which 
they  could  choose.  For  instance,  several  participants  selected  "managed  care"  when 
asked  to  choose  a  single  plan,  apparently  failing  to  recognize  that  all  four  plans 
described  were  managed  care  plans.  As  a  result,  we  developed  introductory 
instructions  and  a  better  overview  that  described  the  choice  task  and  how  to  find 
information  about  the  specific  health  plans  available.  We  also  developed  a  general 
orientation  to  help  consumers  see  the  relationship  between  managed  care  in  general 
and  the  specific  managed  care  health  plans  described  in  the  handbook.  The  first  few 
pages  of  the  handbook  were  modified  to  provide  an  overview  of  the  handbook,  its 
assumptions,  and  its  organization  to  increase  consumers'  motivation  for  using  it. 

Ordering  of  Materials.  Textual  and  graphical  comparison  charts  had  been 
placed  in  the  middle  of  the  brochure,  ahead  of  most  of  the  text  used  to  explain  them. 
Since  there  was  a  strong  tendency  on  the  part  of  enrollees  to  review  the  materials  page 
by  page,  they  reviewed  the  charts  prior  to  the  text  describing  how  to  interpret  them. 
Therefore,  we  moved  the  comparison  charts  to  the  end  of  the  handbook  in  the  next 
version. 

Need  for  Navigational  Aids.  Enrollees  also  had  difficulty  using  the  comparison 
charts  developed  in  the  first  draft  set  of  materials.  Therefore,  we  modified  the  graphs  to 
use  blocks  of  color  and  other  graphical  aids  to  make  the  comparison  chart  easier  to 
read.  We  also  had  included  a  total  of  twelve  performance  measures.  This  number 
seemed  to  overwhelm  participants,  with  a  few  indicating  they  would  probably  skip  over 
most  of  the  graphs  if  they  were  using  the  materials  on  their  own.  Based  on  these 
results,  we  reduced  the  number  of  measures  to  eight  in  the  next  draft  of  the  materials. 

Performance  Measure  Comprehension.  Enrollees  seemed  to  recognize  that 
performance  measures  convey  relatively  complex  information.  Some  thought  these 
indicators  were  "health  messages"  about  what  informed  consumers  have  to  do  to 
receive  the  care  they  need.  As  with  the  focus  groups,  several  participants  suggested 
that  the  indicators  (e.g.,  immunization  rates  and  low  birth  weight  rates)  reflect  plan 
member  behavior  rather  than  the  quality  of  care  received  through  the  plan.  These 
interpretations  often  led  to  testing  responses  in  the  opposite  direction  of  what  might  be 
expected.  For  example,  one  participant  commented  that  the  plan  with  a  high  number  of 
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low  birth  weight  babies  probably  did  a  very  good  job  at  taking  care  of  these  babies 
because  they  had  experience  in  this  area.  As  another  example,  participants  interpreted 
plan  differences  in  late-stage  cancer  diagnosis  as  an  indication  that  the  plans  had 
different  numbers  of  patients  with  cancer. 

The  Medicaid  test  population  used  several  strategies  to  simplify  the  choice  task. 
One  common  strategy  was  selecting  a  single  plan  early  in  the  information  review 
process  based  on  factors  having  most  relevance  for  that  individual's  personal 
circumstances.  Individuals  using  this  approach  would  then  selectively  use  other 
information  to  justify  their  initial  choice.  Based  on  these  findings,  we  revised  the 
performance  measure  sections  to  include  more  text  that  clearly  explained  how  to 
interpret  the  measures  and  what  the  differences  meant  for  plan  members. 

Other  Handbook  References.  Medicaid  beneficiaries  seemed  to  like  the 
magazine  format  of  the  handbook  and  appreciated  the  photographs.  Some 
misunderstood  the  consumer  satisfaction  measures;  their  confusion  stemmed  in  part 
failure  to  understand  the  survey  sampling  processes  and  representativeness. 
Participants  indicated  they  were  very  interested  in  information  about  plan  review  by 
outside  agencies  such  as  the  National  Committee  on  Quality  Assurance  (NCQA). 
Several  participants  wanted  more  detail  regarding  NCQA  (e.g.,  was  it  a  government 
agency,  an  independent  agency,  or  a  consumer  group).  We  modified  our  description 
accordingly  for  the  next  version  of  the  handbook. 

Lessons  Learned  from  the  Second  Round  of  Cognitive  Testing 

Usefulness  of  Handbook.  Participants  unanimously  agreed  they  would  like  to 
have  something  like  the  Medicaid  HMO  Handbook  if  they  were  choosing  an  HMO, 
primarily  because  they  thought  it  would  make  choosing  a  plan  easier.  When  asked 
what  part  of  the  handbook  they  found  most  useful,  the  most  commonly  mentioned 
section  was  the  "Performance  Measure  Graphs,"  mentioned  by  five  participants.  When 
asked  what  parts  of  the  booklet  were  least  helpful  to  them,  many  (6)  participants  said 
that  everything  was  important. 

Understanding  Terminology.  Seven  of  the  nine  testing  participants  referred  to 
a  primary  care  provider  (PCP)  in  some  way  as  their  "own  doctor"  or  "the  doctor  who 
sees  you  first  for  all  kinds  of  care."  Although  most  of  the  Medicaid  beneficiaries 
understood  the  term  PCP,  we  elected  to  use  the  term  "primary  care  doctor." 

After  reading  the  "What  about  Emergencies?"  section,  about  half  (5)  of  the 
participants  were  able  to  distinguish  between  "urgent  care"  and  "emergency  care" 
based  on  their  reading  of  the  handbook.  Two  participants  distinguished  between  the 
two  types  of  care  but  in  the  wrong  direction,  describing  urgent  care  as  more  serious 
than  emergency  care.  The  other  two  participants  said  they  were  unable  to  separate  the 
two  types  of  care  because  for  both  you  need  care  "right  then  and  there."  Based  on 
these  findings,  we  made  further  attempts  to  revise  the  information  in  this  section. 
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Comprehension  of  Performance  Measures.  In  the  second  round  of  testing  we 
modified  the  protocol  to  test  how  well  Medicaid  enrollees  were  able  to  use  the 
performance  measures.  In  developing  the  handbook,  we  had  intentionally  manipulated 
the  performance  measure  graphics  so  that  one  HMO  was  clearly  superior  to  the  others 
based  on  the  data.  Our  test  results  suggested  the  changes  in  the  layout  (i.e.,  placing 
this  information  at  the  end  of  the  handbook,  improved  graphics,  and,  perhaps  most 
important,  reducing  the  number  of  indicators  from  twelve  to  eight)  enhanced  the 
beneficiaries'  ability  to  make  plan  comparisons.  After  reviewing  the  materials,  all  nine 
participants  selected  "Mednet,"  the  HMO  that  performed  the  best  on  all  eight 
performance  measures.  The  ability  to  use  the  comparative  information  was  further 
tested  through  the  use  of  two  vignettes.  Seven  of  the  eight  respondents  receiving  the 
first  vignette  identified  the  appropriate  plan  for  the  story  character;  six  of  eight 
responded  appropriately  to  the  second  vignette. 

Participants  were  specifically  asked  what  part  of  the  handbook  they  relied  on 
most  to  choose  their  plan.  Almost  all  (8)  referred  to  either  the  performance  measure 
graphs  or  to  both  the  graphs  and  the  comparison  chart  (5).  Seven  of  the  nine 
described  the  graphs  an  easy  to  use.  Based  on  the  heavy  reliance  on  the  graphs,  we 
added  a  section  at  the  end  of  the  booklet  to  remind  people  to  look  back  at  both  the 
charts  and  the  graphs,  rather  than  relying  only  on  the  graphs  (which,  in  the  current 
layout,  are  at  the  end). 

Based  on  the  tendency  to  treat  small  plan  differences  as  meaningful,  we  added 
an  explanation  of  the  concept  of  "margin  of  error"  to  the  Medicaid  handbook  in  the 
performance  measure  section.  In  developing  our  materials,  we  were  unable  to  locate  a 
satisfactory  explanation  of  this  concept  that  we  thought  was  simple,  accurate,  and 
appropriate  for  our  target  populations.  After  much  deliberation,  we  inserted  a  sentence 
explaining  to  users  that  differences  of  less  than  10%  may  not  be  meaningful.  In 
addition,  we  provided  examples  of  when  differences  were  and  were  not  statistically 
significant.  We  had  not  included  this  explanation  prior  to  the  second  round  of  cognitive 
testing  so  this  text  remains  untested.  Explanation  of  statistical  differences  remains,  in 
our  opinion,  is  an  area  warranting  a  great  deal  of  additional  investigation. 

Performance  Measure  Preferences.  When  askPd  to  choose  the  performance 
measures  they  considered  most  important,  "pediatric  immunizations"  and  "preventable 
hospitalizations  for  asthma"  were  the  indicators  most  often  mentioned,  followed  closely 
by  "member  satisfaction  with  doctors'  communication."  The  indicator  mentioned  most 
frequently  as  least  relevant  was  "prenatal  care."  However,  we  recommend  avoiding 
utilization  measures  that  relate  to  ambulatory  sensitive  conditions,  such  as 
hospitalizations  for  asthma.  Lower  hospitalization  rates  for  ambulatory  sensitive 
conditions  unambiguously  reflect  higher  quality  in  a  fee-for-service  setting.  Under 
capitation,  however,  quality  as  a  cause  for  low  hospitalization  rates  is  confounded  with 
the  HMO's  incentive  to  limit  utilization  for  financial  reasons. 

Comparison  Chart  Preferences.  Most  (7)  of  the  beneficiaries  thought  the  plan 
comparison  chart  was  "very  useful."  When  asked  which  aspect  of  the  chart  was  most 
useful  to  them,  responses  varied,  with  the  most  frequently  mentioned  feature  being  the 
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"special  features"  section  which  indicates  whether  doctors'  offices  are  on  the  bus  routes 
and  availability  of  weekend  hours.  The  least  helpful  piece  of  information  was  the 
section  on  languages  used.  This  may,  however,  be  important  for  those  with  limited 
English  ability,  as  was  shown  in  recent  AHCPR  focus  groups. 

Six  of  the  nine  respondents  reported  that  knowing  the  NCQA  rating  of  a  plan 
made  no  difference  to  them  in  choosing  a  plan.  In-depth  probing  suggested  this 
information  actually  confused  participants.  One  person,  for  example,  believed  it 
represented  the  status  of  the  doctors  in  the  plan,  rather  than  the  plan  itself,  remarking 

"How  can  you  go  to  a  doctor  who  has  a  one-year  accreditation? 
I  wouldn't  go  to  that  one!" 

In  the  end,  we  deleted  the  section  of  the  comparison  chart  showing  each  plan's  NCQA 
rating,  as  there  are  multiple  accreditation  bodies,  and  we  did  not  want  the  handbooks  to 
appear  to  promote  one  over  another. 

We  also  uncovered  a  potential  concern  that  should  be  considered  in  developing 
all  future  materials.  Several  of  the  Medicaid  enrollees  reported  that  they  assumed  that 
a  plan's  inclusion  in  the  handbook  meant  that  the  plan  had  already  met  certain 
standards  of  excellence.  A  couple  of  participants  expressed  it  this  way, 

"I  figure  if  the  plan  was  listed  in  this  booklet,  it  already  has  a 
quality  rating." 

"I'm  quite  sure  if  they  didn't  approve  it,  they  wouldn't  be 
offered." 

Thus,  while  it  is  important  to  communicate  information  to  consumers  about  the  choices 
available,  it  also  is  important  to  indicate  whether  or  not  these  plans  have  been 
preliminarily  screened. 

4.2.2  Medicare  Materials 

4.2.2.1  Medicare  HMO  Video 

Beneficiaries  generally  reported  that  the  videotape  was  easy  to  understand  and 
easy  to  watch.  Several  participants  noted  that  the  comparisons  between  grocery 
shopping  and  shopping  for  health  care  seemed  appropriate  to  them.  There  were  no 
reports  of  difficulties  understanding  terminology  used  in  the  videotape.  A  few  partici- 
pants indicated  the  video  "talked  down"  to  them  in  a  few  places.  Some  understood  that 
the  videotape's  tone  may  be  necessary  for  a  general  Medicare  audience  with  different 
types  and  levels  of  experiences. 

We  experienced  some  of  the  same  difficulties  discussed  earlier  concerning  the 
Medicare  HMO  video.  First,  the  cost  of  producing  the  video  was  not  trivial.  Second, 
edits  after  the  fact  were  very  limited.  Because  a  "realistic"  setting  was  used,  substantial 
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edits  in  the  script  could  only  be  made  through  reshooting,  potentially  doubling  the 
production  costs.  Edits  made  to  the  inserted  graphics  were  more  manageable.  Again, 
we  believe  it  would  have  been  beneficial  to  produce  the  video  as  the  final  set  of 
materials,  drawing  upon  the  lessons  learned  through  the  cognitive  testing  of  the  other 
materials.  Additionally,  we  would  have  allocated  a  greater  percentage  of  the  budget 
and  schedule  for  reshooting  as  appropriate. 

4.2.2.2  Medicare  HMO  Handbook 

Lessons  Learned  from  the  First  Round  of  Cognitive  Testing 

Comprehension  and  Navigation.  Testing  results  showed  that  numerous 
changes  to  the  materials  were  necessary.  Although  most  beneficiaries  recognized  they 
were  being  asked  to  choose  among  competing  HMOs,  several  failed  to  recognize  that 
the  text  and  graphical  comparison  charts  were  providing  comparative  information  on 
four  HMOs.  Several  beneficiaries  were  confused  about  how  Medicare  HMOs  compare 
to  standard  Medicare  and  Medigap  coverage.  Thus,  we  added  an  introductory  section 
on  the  different  fee-for-service  and  managed  care  supplemental  options  available  to 
Medicare  beneficiaries  to  help  orient  beneficiaries.  Some  beneficiaries  were  suspicious 
about  HMOs  and  perceived  that  the  initial  materials  seemed  to  promote  them.  In 
response,  we  edited  the  handbook  to  present  a  more  balanced  perspective.  Further,  in 
the  original  version  of  our  HMO  handbook,  in  our  efforts  to  layer  materials,  we  placed 
the  performance  measure  graphs  in  the  brochure  as  a  centerfold.  Unfortunately,  this 
layout  placed  the  graphs  ahead  of  most  of  the  text  explaining  how  to  interpret  them, 
causing  participants  to  have  difficulty  understanding  them.  In  addition,  in  the 
observation-only  interviews,  several  participants  failed  to  look  at  this  information.  In  the 
revised  materials,  the  graphs  were  placed  at  the  back  of  the  brochure.  Participants 
apparently  recognized  that  selected  indicators  conveyed  relatively  complex  information, 
but  some  interpreted  these  as  "health  messages"  about  what  informed  consumers  had 
to  do  to  receive  the  care  they  need. 

Lessons  Learned  from  the  Second  Round  of  Cognitive  Testing 

As  previously  indicated,  the  first  round  of  cognitive  testing  provided  very 
insightful  and  valuable  data  for  revising  the  prototype  materials.  The  second  round  of 
testing  re-evaluated  the  content,  navigation,  comprehension,  and  consumer  decision 
processes  for  these  revised  materials. 

Overall  Usefulness.  Beneficiaries  participating  in  testing  unanimously  agreed 
they  would  like  to  have  a  booklet  like  the  Medicare  HMO  Handbook  if  they  were 
choosing  an  HMO.  Reasons  offered  include  that  it 

•  is  concise 

•  is  a  resource  that  can  be  revisited 

•  focuses  on  questions  one  should  ask  before  choosing  an  HMO  and 
is  broken  down  into  understandable  sections 
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•  is  clear  about  its  purpose  from  the  beginning 

•  provides  a  helpful  comparison  chart. 

After  reading  the  entire  handbook,  beneficiaries  were  asked  which  plan  they 
would  pick  if  they  had  to  choose  one.  All  nine  were  able  to  pick  a  plan,  although  no 
plan's  data  were  constructed  to  make  it  the  "best"  choice  as  with  the  Medicaid 
Handbook  (as  had  been  the  case  for  the  Medicaid  testing).  Participants  generally 
focused  on  the  comparison  chart  information  to  make  their  choices. 

The  most  useful  part  of  the  handbook  according  to  participants  was  the 
comparison  chart.  The  least  helpful  section  was  "Voices  about  Choices,"  which  was 
subsequently  deleted  from  the  final  version  of  the  handbook. 

Comprehension  and  Navigation.  After  the  first  round  of  testing,  substantial 
changes  were  made  to  improve  navigation.  A  great  deal  of  effort  was  expended  on 
better  linking  the  various  sections,  improving  the  flow  from  one  section  to  another.  Our 
results  for  the  second  round  of  testing  suggested  the  handbook  worked  much  more 
effectively  as  a  whole,  with  only  minor  changes  needed  in  specific  sections. 

Understanding  Terminology.  Seven  of  the  nine  beneficiaries  for  the  second 
round  of  cognitive  testing  were  confused  or  felt  others  would  be  confused  by  the  term 
HMO.  The  handbook  also  included  an  explanation  of  risk  versus  cost  HMOs  and  the 
implications  of  these  different  managed  care  options  on  out-of-pocket  costs.  Eight  of 
the  participants  understood  the  concept  of  point-of-service  plan  but  did  not  understand 
how  it  compared  to  risk  versus  cost  options.  As  a  result  of  this  confusion,  we  attempted 
to  clarify  these  concepts  in  the  final  version  of  the  handbook  by  adding  a  comparison 
chart  on  page  5. 

We  specifically  tested  beneficiaries'  preferences  and  "lay"  understanding  of  the 
terms  "provider"  and  "doctor."  There  was  almost  unanimous  preference  for  the  te  rn 
doctor.  When  asked  by  the  cognitive  tester  whether  "provider"  means  a  "person,  a 
place,  or  both,"  almost  every  participant  said  "provider"  refers  to  a  person  rather  than  a 
place  and  many  indicated  that  provider  could  include  health  care  professionals  other 
than  a  doctor.  One  beneficiary  saw  Medicare  as  the  "provider,"  denoting  a  provider  of 
insurance  rather  than  care.  Based  on  these  testing  results,  we  elected  to  use  the  term 
"doctor  and  other  health  care  professionals"  for  provider  in  the  final  version  of  the 
Medicare  Managed  Care  handbook. 

Most  beneficiaries  had  heard  the  term  "primary  care  provider"  (PCP)  before 
reading  the  handbook.  Similar  to  the  Medicaid  testing  results,  the  majority  referred  to 
PCP  as  their  doctor  or  the  doctor  who  sees  you  first  for  care.  One  participant  saw 
Medicare  as  the  primary  provider  and  the  insurance  company  as  the  secondary 
provider.  To  avoid  confusion,  we  elected  to  use  "primary  care  doctor"  in  the  final 
version  of  the  handbook  since  participants  used  this  term  to  refer  to  the  main  person 
who  gives  them  care. 
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4.2.3  Medicare  Options  Handbook 

In  this  section,  we  summarize  the  findings  from  the  cognitive  testing  interviews 
with  60-64  year-olds,  which  examined  the  Medicare  Options  handbook  and  worksheets. 

Lessons  Learned  from  the  First  and  Second  Round  of  Cognitive  Testing: 
General  Issues 

Overall  Usefulness.  In  general,  the  Medicare  Options  handbook  was  well 
received.  Responses  were  more  positive  during  the  second  round  of  testing  after 
significant  revisions  had  been  made  in  response  to  round  one  findings.  Participants 
indicated  that  they  would  use  the  handbook  if  it  were  sent  to  them.  Some  participants 
mentioned  the  relevance  of  the  booklet  to  their  particular  age  group.  That  is,  while  the 
booklet  would  not  have  interested  them  a  couple  of  years  ago,  it  does  now  because 
they  are  nearing  the  time  when  they  need  to  begin  thinking  about  Medicare.  The 
sections  most  commonly  mentioned  by  participants  as  being  most  useful  were: 
"Step  4:  The  Benefits  and  Cost  of  Medigap  Insurance"  and  "Understanding  Your 
Medicare  Options,"  which  contains  pie  chart  graphics  depicting  how  different  sources 
(e.g.,  Medicare,  supplemental  insurance,  and  individual  out-of-pocket  costs)  contribute 
to  the  payment  of  health  care  services. 

Comprehension  and  Navigation.  Testing  results  showed  that  the  handbook 
was  effective  at  assisting  people  in  choosing  an  option  as  measured  by  the  fact  that 
most  participants  were  able  to  make  a  choice  between  the  options  when  asked  to  do 
so.  Participants  were  evenly  split  between  a  Medigap  plan  and  a  Medicare  HMO,  with  a 
limited  number  selecting  an  employer  plan.  Because  participants  tended  to  prefer  the 
more  comprehensive  benefit  packages  (especially  in  the  case  of  Medigap) — which  are 
in  reality  purchased  by  only  a  small  portion  of  beneficiaries  because  of  their  high 
cost — it  seemed  necessary  to  stress  the  importance  of  making  trade-offs  between  what 
a  person  would  like  and  what  is  affordable.  Following  the  benefits  charts  on  pages  16 
and  19,  we  added  text  emphasizing  the  need  to  make  realistic  trade-offs  between  the 
"benefits  you  want  and  what  you  can  afford." 

Based  on  testing  results  in  both  rounds,  additional  navigational  guidance  was 
added  throughout  the  handbook.  In  particular,  respondents  indicated  that  they  needed 
further  clarification  regarding  how  to  read  the  charts  and  use  the  worksheets.  More 
introductory  text  was  needed  at  the  outset  of  the  worksheet  section  to  inform  readers  of 
the  process  they  are  about  to  undertake.  Specifically,  we  explained  that  there  is  a 
section  of  the  worksheet  for  each  supplemental  option.  And,  within  each  option, 
readers  have  an  opportunity  to  review  the  covered  benefits  offered  by  each  plan  and 
calculate  the  total  costs  for  the  plans  of  interest  to  them.  Finally,  the  summary 
worksheet  will  ask  them  to  review  the  options  and  the  individual  plans  one  more  time 
before  making  a  choice.  Persons  who  had  already  made  their  choice  by  this  time  were 
told  to  "opt  out"  of  this  task. 
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Lessons  Learned  from  the  First  and  Second  Rounds  of  Cognitive  Testing: 


Specific  Sections 


Definitions/Examples.  Because  some  participants  mentioned  that  certain 
terms  used  in  the  handbook  (e.g.,  "Medicare  approved  charge,"  "durable  medical 
equipment,"  and  "home  health  care")  were  unclear  to  them,  we  defined  these  terms  in 
the  glossary  or  added  examples  when  mentioned  in  the  text  to  enhance  comprehen- 
sion. We  felt  that,  although  these  terms  confused  people,  they  needed  to  be  included 
to  provide  an  accurate  description  of  Medicare  and  supplemental  insurance  coverage. 

Recognizing  the  Different  Options  and  Comprehending  the  Pie  Charts. 

Because  our  focus  groups  indicated  at  least  some  new  beneficiaries  Medicare 
assumed  Medicare  would  cover  all  their  health  care  needs,  we  developed  "pie  charts" 
to  depict  the  proportion  of  health  care  costs  Medicare  in  fact  does  cover.  Most 
cognitive  testing  participants  recognized  that  the  pie  charts7  graphically  depict  how 
different  financial  sources  contribute  to  the  payment  of  health  care  services.  In 
developing  this  section  of  the  handbook,  extensive  care  was  taken  to  show  this 
relationship  objectively  and  accurately.  This  was  a  challenge  because  the  distribution 
of  payment  is  largely  a  function  of  the  specific  health  plan  and  the  health  status  and 
utilization  level  of  the  individual.  Participants  were  generally  in  favor  of  using  pie  charts 
to  show  this  relationship. 

The  proportion  of  the  pie  chart  on  page  4  reflecting  how  much  Medicare  pays  for 
health  care  expenditures  is  based  on  published  data.  On  average,  Medicare  pays  for  45 
percent  of  total  health  care  expenditures  for  persons  age  65  and  over  (PPRC,  1996),8 
with  the  balance  of  the  costs  being  the  responsibility  of  the  beneficiary.  Since  these 
costs  may  be  extensive,  most  individuals  purchase  some  type  of  supplemental 
insurance  if  they  do  not  qualify  for  Medicaid.  One  purpose  of  the  pie  chart  on  page  4 
was  to  show  how  costs  are  divided  among  Medicare  and  the  beneficiary.  Another 
purpose  was  to  explain  that,  even  with  supplemental  insurance,  beneficiaries  may  still 
face  out-of-pocket  costs.  The  proportion  of  costs  not  covered  by  Medicare  (55  percent) 
is  divided  evenly  in  the  pie  chart  (between  the  supplemental  insurer  and  the  beneficiary) 
for  simplicity,  but  it  is  not  empirically  based. 

We  explicitly  decided  not  to  show  the  numerical  proportions  of  each  pie  piece,  to 
discourage  readers  from  thinking  that  this  was  exactly  how  expenses  would  be  divided 
among  payment  sources.  Despite  this,  some  cognitive  testing  respondents  still  made 
this  incorrect  assumption.  Thus,  we  added  text  to  page  5  stating  that:  The  exact 
amount  you  pay  will  vary  based  on  the  option  you  choose.  It  depends  on:  (1)  the  type 
of  plan  you  pick,  (2)  the  benefits  it  covers,  and  (3)  the  health  care  services  you  use. 
We  also  informed  readers  that  completing  the  worksheets  would  give  them  an  even 


7  In  a  earlier  version  of  the  handbook,  there  were  multiple  pie  charts. 

8  This  proportion  factors  in  payment  for  long-term  care  nursing  home  expenditures,  which 
usually  are  not  covered  by  Medicare. 
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better  idea  of  how  much  they  could  expect  to  pay  out-of-pocket  annually.  We 
emphasized,  however,  that  even  these  calculations  should  be  considered  estimates. 

After  the  second  round  of  testing,  we  reduced  the  number  of  pie  charts  in 
response  to  outside  reviewer  comments  received  on  the  materials.  In  general,  there 
was  concern  that  having  multiple  pie  charts  gave  the  incorrect  impression  that  each 
supplemental  option  covered  the  same  amount  of  expenses  (because  the  portion  of  the 
pies  reflecting  supplemental  insurance  was  consistent  across  the  options). 

Understanding  the  Different  Fee-for-Service  and  Managed  Care  Options. 

Immediately  following  the  introductory  section  containing  the  pie  charts,  the  handbook 
provides  an  overview  of  each  supplemental  insurance  option  as  well  as  "Medicare  only" 
coverage.  One  full  page  is  dedicated  to  Medigap  and  another  to  Medicare  HMOs 
because  of  the  amount  of  information  that  needs  to  be  conveyed  about  these  two 
options.  We  learned  that  readers  were  confused  between  Medicare  Part  A  and  Part  B 
and  Medigap  plans  A  through  J.  Minor  changes  were  made  in  the  Medigap  page  to 
alleviate  this  problem.  When  tested,  respondents  indicated  that  the  Medigap  and 
Medicare  HMO  sections  were  "overloaded"  with  information  and  were  in  need  of  more 
user-friendly  layout.  In  response,  we  added  as  much  "white  space"  as  possible,  used 
several  text  boxes  in  the  margins  to  diversify  how  the  material  is  presented,  and 
removed  all  photos. 

Cognitive  testing  participants  also  had  some  difficulty  understanding  how 
Medicare  SELECT  plans  differ  from  standard  Medigap  plans,  how  point-of-service 
plans  differ  from  Medicare  HMOs,  and  how  SELECT  plans  differ  from  point-of-service 
HMOs.  As  a  result,  we  added  a  new  page  (page  9)  to  the  Options  handbook  containing 
a  chart  summarizing  the  key  differences  between  the  Medicare/Medigap  and  Medicare 
HMO  options  across  the  following  dimensions:  choice  of  providers,  access  to  specialty 
care,  geographic  availability  of  services,  benefits,  and  premiums.  The  goal  was  to 
provide  a  quick  visual  overview  of  .the  some  of  the  key  factors  considered  when 
choosing  between  options. 

Based  on  comments  received  from  outside  reviewers,  we  revised  the  sections  of 
the  handbook  addressing  employer-sponsored  supplemental  insurance  (pages  6  and 
11).  The  version  tested  in  Round  Two  of  the  interviews  did  not  clearly  specify  that 
whether  Medicare  is  the  primary  or  secondary  payer  depends  on  several  factors  (e.g., 
whether  the  beneficiary  or  his/her  spouse  is  still  working  and  the  size  of  the  employer). 
Therefore,  the  text  was  changed  to  address  this.  Finally,  to  address  some  of  the 
commonly  asked  questions  raised  during  cognitive  testing,  we  added  information  about 
how  one  supplemental  policy  is  generally  sufficient. 

Self-Defined  Health  Care  "User  Status"  and  Out-of-Pocket  Expenses.  A 

unique  aspect  of  the  Medicare  Options  handbook  is  its  ability  to  help  people  calculate 
an  annual  estimate  of  out-of-pocket  health  care  expenses  they  may  incur  with  different 
types  of  supplemental  options.  To  do  this,  readers  are  asked  to  first  classify 
themselves  as  a  "high,"  "medium,"  or  "low"  user  of  services.  A  description  of  each  "user 
status"  level  is  provided  to  enable  readers  to  decide  how  they  should  categorize 
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themselves.  The  descriptions  show  how  much  health  care  services  on  average  people 
in  each  user  category  consume  per  year  in  terms  of  doctor  visits,  hospitalizations,  and 
prescription  drugs.  In  response  to  reviewer  comments,  we  added  more  detailed 
information  (average  number  of  home  health  visits  per  year)  to  help  readers  classify 
themselves  as  a  medium  or  high  user,  because  the  utilization  levels  of  the  two  groups 
were  similar.  We  reminded  readers  that  health  care  utilization  is  generally  positively 
correlated  with  age.  Because  we  found  that  beneficiaries  tended  to  assign  themselves 
on  the  basis  of  their  current  health  status,  rather  than  anticipating  future  needs  for  care, 
we  added  a  statement  encouraging  them  to  consider  their  future  needs  when  assigning 
themselves  to  a  category.  We  also  recommended  that  those  who  are  uncertain  about 
their  health  care  use  should  choose  the  higher  level. 

This  approach  builds  on  the  work  of  Sofaer,  Kenney  and  Davidson  (1992),  who 
computed  out-of-pocket  expense  estimates  for  Medicare  beneficiaries  with  13  different 
medical  conditions  under  several  different  Medigap  and  Medicare  HMO  options 
available  in  Los  Angeles  before  OBRA  1990  Standardization.  Our  cognitive  research 
suggested  that  beneficiaries  would  not  identify  themselves  with  one  of  these  groups  if 
they  did  not  have  the  condition.  Thus,  we  classified  persons  in  the  more  general 
categories  of  high,  medium,  or  low  users  of  health  care.  All  cognitive  interview 
participants  were  easily  able  to  assign  themselves  to  one  of  these  categories. 

Beneficiaries  employ  the  user-level  status  category  later  in  the  worksheet  to 
estimate  their  expected  out-of-pocket  expenses  under  different  insurance  options. 
Most  beneficiaries  understood  that  these  estimates  were  averages  and  that  their  actual 
experience  would  differ.  However,  we  still  felt  it  necessary  to  add  the  statement  that  the 
out-of-pocket  cost  data  are  only  estimates. 

Benefit  Comparison  Charts.  Response  to  the  benefit  comparison  charts 
varied,  in  part  because  of  individual  preferences  for  the  amount  of  information  desired, 
with  some  participants  wanting  more  detail  and  others  responding  that  the  charts  were 
overloaded.  Stating  only  what  benefits  were  covered  was  not  adequate  for  some;  they 
wanted  the  chart  to  state  explicitly  what  benefits  they  will  need  to  pay  for  themselves. 
We  concurred  that  the  benefits  chart  in  the  Guide  to  Health  Insurance  for  People  with 
Medicare  did  this  well  because  it  includes  a  specific  column  of  information  explaining 
what  "You  Pay."  Thus,  we  modified  our  Medicare  Only  chart  as  such  and  it  tested  well. 
Another  example  of  the  variation  across  preferences  is  that  some  readers  felt  that  the 
"Reviewing  Medigap  Benefits"  and  "Reviewing  Medicare  HMO  Benefits"  charts  were 
redundant  with  their  respective  benefit  comparison  charts.  These  charts  were  intended 
to  provide  another  approach  to  reviewing  benefits  by  encouraging  people  to  think  about 
which  services  are  important  to  them.  We  decided  to  keep  the  review  charts  in  the 
handbook  since  they  did  appeal  to  some  respondents. 

Accurately  listing  what  each  supplemental  plan  covers  in  a  limited  amount  of 
space  presented  many  challenges.  In  the  end,  we  found  it  best  to  use  very  brief  but 
consistent  descriptions,  referring  readers  to  the  Guide  for  more  detail  if  needed.  We 
also  made  sure  the  benefit  comparison  chart  in  each  option  section  was  consistent  in 
terms  of  the  order  in  which  the  benefits  were  listed.  In  response  to  comments  received 
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from  both  outside  reviewers  and  during  cognitive  testing,  we  added  benefit  information 
about  dental  services  and  long-term  care  to  each  benefit  chart. 

Cost  Comparison  Charts.  The  greatest  challenge  testing  participants  faced 
when  using  the  Medigap  and  Medicare  HMO  cost  comparison  charts  was  manipulating 
figures.  Most  attributed  this  difficulty  to  having  instructions  that  were  not  clear  enough. 
Following  each  round  of  testing,  changes  were  made  to  the  instructions  based  on  the 
specific  problems  expressed  by  participants.  One  important  change  was  to  move  the 
rows  in  the  charts  so  that  data  which  needed  to  be  summed  together  (premiums  and 
out-of-pocket  costs)  were  adjacent. 

The  charts  were  simplified  in  other  ways  as  well.  For  example,  in  an  earlier 
version  of  the  Medigap  cost  chart,  data  showing  the  range  of  plan  premiums  were 
included  so  beneficiaries  could  see  at  a  glance  which  plans  fall  within  their  budget.  In 
the  end,  we  decided  that  despite  the  fact  that  this  information  might  be  useful,  it 
seemed  to  add  unnecessary  confusion  to  the  chart  and  was  deleted.  Beneficiaries 
must  consult  their  state  premium  guide  for  this  information. 

Summary  Chart.  The  summary  chart  on  page  21  underwent  a  significant 
metamorphosis  throughout  the  project.  As  previously  mentioned,  the  purpose  of  the 
chart  is  to  allow  readers  to  reconsider  all  the  options  before  making  a  choice.  It 
enables  readers  to  look  at  plan-specific  data  (total  cost,  benefits,  doctor  choice)  across 
the  options  in  one  location.  This,  however,  requires  that  readers  must  move  their 
calculations  and  ratings  from  previous  pages  to  the  summary  chart.  This  posed 
difficulty  for  some  and  not  for  others.  Again,  the  instructions  were  simplified  as  much 
as  possible  and  extraneous  information  was  removed  from  the  chart.  The  chart  leaves 
space  in  rows  five  and  six  for  readers  to  write  in  individual  comments  regarding  "things  I 
like"  and  "things  I  don't  like"  to  round  out  their  decision  making. 

At  the  conclusion  of  the  page,  readers  are  congratulated  on  completing  the 
exercise.  They  are  encouraged  to  seek  additional  information  or  assistance  in  their 
decision  making  process. 

Lessons  Learned  from  the  Third  and  Fourth  Rounds  of  Cognitive  Testing: 
General  Issues 

Overall  Response  to  the  Handbook.  All  24  round  three  and  four  testing 
participants  agreed  that  they  would  use  the  Medicare  Options  handbook  if  it  were  sent 
to  them  at  home,  citing  that  it  was  useful  in  helping  them  to  make  a  health  plan  choice. 
Participants  rated  both  the  handbook  and  the  worksheet  section  positively  as  relatively 
easy  to  use  and  causing  limited  frustration  and  confusion.  In  general,  participants  felt 
that  the  handbook  would  "get  them  started,"  but  that  additional  resources  would  be 
needed  in  order  to  come  to  a  decision.  Several  participants  said  they  would  like  to  have 
a  counselor  available  to  whom  they  could  address  questions. 

Differences  in  Comprehension  and  Navigation  According  to  Education 
Level.  Overall,  results  from  the  higher  and  lower  educated  groups  in  round  four  were 
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very  comparable  across  nearly  all  measures.  This  finding  suggests  that  the  handbook 
may  be  simple  enough  to  be  understood  by  those  with  at  least  a  high  school  education, 
but  larger  scale  testing  would  be  needed  to  have  greater  confidence  in  this  result. 
Some  variation  in  comprehension  existed  within  the  lower  educated  group;  a  couple  of 
the  participants  clearly  mastered  the  information  better  than  others.  Regardless  of 
educational  status,  Medicare  beneficiaries  may  require  assistance  to  fully  understand 
the  material  and  are  likely  to  seek  this  assistance  and/or  more  information  before 
making  a  health  plan  decision. 

Lessons  Learned  from  the  Third  and  Fourth  Rounds  of  Cognitive  Testing: 
Specific  Sections 

Comprehension  of  the  Pie  Chart  Message.  Participants  were  able  to  explain 
correctly  the  intended  purpose  of  the  pie  chart  section,  stating  in  their  own  words  that 
the  pie  chart  represents  three  different  sources  of  payment  for  health  care  services. 
When  asked  in  round  three,  11  out  of  12  participants  acknowledged  that  Medicare  does 
not  cover  all  health  care  costs;  only  about  a  third  of  this  group  also  noted  that  all  health 
care  costs  may  not  be  covered  even  with  a  supplement.  Minor  changes  were  made 
after  round  three  to  pages  4  and  5  to  enhance  readability. 

Medigap  Open  Enrollment  Period.  In  the  round  four  survey,  participants 
recalled  reading  about  the  Medigap  open  enrollment  period.  When  asked  to  describe  it 
briefly,  participants  generally  noted  correctly  that  it  is  a  one-time  only  period  once  you 
enroll  in  Medicare  when  you  cannot  be  turned  down  by  a  plan  regardless  of  your 
medical  history.  This  seemed  to  be  an  improvement  over  the  level  of  comprehension 
expressed  in  round  three  during  which  only  8  of  12  participants  noted  that  the  period  is 
only  six  months  long  and  only  half  (6)  noted  that  it  is  only  a  one-time  occurrence. 

Differences  Between  FFS  and  Managed  Care  Options.  When  asked  to 
describe  the  main  differences  when  FFS  and  managed  care  options  in  round  three, 
every  participant  noted  that  the  premium  costs  differ.  Many  also  acknowledged  the 
restriction  of  choice  of  doctors,  but  fewer  noted  other  key  aspects  of  HMOs.  Again  in 
round  four,  participants  most  commonly  mentioned  doctor  choice/selection  and  price, 
correctly  seeing  the  two  as  a  trade-off.  However,  a  few  of  the  lesser  educated 
participants  were  not  able  to  articulate  the  difference  well.  They  also  found  page  nine, 
which  contains  a  comparison  chart  of  the  different  types  of  FFS  and  managed  care 
plans  for  Medicare  beneficiaries,  to  be  confusing.  We  expect  that  a  chart  containing 
that  amount  of  information  will  require  more  time  than  was  available  in  the  testing 
situation  to  be  more  fully  understood. 

Selecting  User  Level  Status.  Nine  participants  in  the  third  round  and  1 1 
participants  in  the  fourth  round  of  testing  circled  a  user  level  status  on  page  12.  In  each 
group,  only  two  participants  said  they  had  some  difficulty  choosing  a  single  level.  Nearly 
all  choose  either  a  low  or  medium  user  level. 

Cost,  Benefit,  and  Summary  Charts.  In  round  three,  participants  were  asked 
to  interpret  the  benefit  charts.  Most  gave  correct  responses,  suggesting  they  were  able 
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to  understand  the  charts.  In  round  three,  nine  participants  completed  the  Medigap  cost 
worksheet  and  seven  completed  the  Medicare  HMO  cost  worksheet.  Following  Round 
Three,  we  changed  the  cost  charts  and  the  summary  chart  to  a  landscape  format  so 
that  the  instructions  for  each  row  could  be  placed  directly  to  the  left  side  of  each 
respective  row.  Row  instructions  were  further  simplified.  By  round  four,  virtually  every 
participant  filled  out  all  the  cells  of  the  cost  worksheets.  Of  the  1 1  participants  in  round 
three  who  completed  the  summary  worksheet,  eight  said  it  was  "very  useful."  All  12 
participants  were  able  to  complete  the  summary  chart  in  round  four. 

4.2.4  Summary  of  Cognitive  Testing  Results 

Results  from  our  final  rounds  of  testing  suggest  that  the  revised  versions  of  the 
materials  are  substantially  more  user-friendly  and  effective  at  communicating  complex 
choice  information  than  the  predecessor  drafts.  We  were  pleased  at  the  degree  to 
which  users  reported  they  found  the  materials  helpful,  as  well  as  the  extent  to  which 
they  could  successfully  use  the  information  to  choose  a  plan.  There  are  still  some 
areas  that  still  need  to  be  addressed  before  widespread  use  of  the  materials.  For 
example,  our  new  text  designed  to  help  consumers  estimate  statistical  differences  will 
need  to  be  tested,  both  in  terms  of  consumers'  ability  to  understand  it  and  the 
appropriateness  of  the  size  of  the  difference  we  used. 

4.2.5  Revising  the  Materials  for  Future  Use 

It  is  also  important  to  recognize  that  the  data  on  which  the  prototypes  are  based 
will  need  to  be  modified  and  tailored  to  the  specific  region  in  which  they  would  be  used. 
Obviously,  the  specific  plans  will  need  to  reflect  plans  available  in  a  given  region.  In 
addition,  as  Medicare  and  Medicaid  guidelines  continue  to  evolve,  these  changes  will 
need  to  be  incorporated.  The  intent  in  designing  those  particular  prototypes  was  to 
develop  user-friendly  prototypes  which  could  be  easily  modified  to  meet  the  needs  of 
multiple  users.  In  some  instances,  the  user  may  choose  to  only  incorporate  specific 
sections.  To  facilitate  the  process  of  adapting  the  materials  to  the  specific  requirements 
of  a  given  situation,  the  print  files  have  been  given  to  HCFA  and  the  print  specifications 
follow  in  Exhibit  4. 
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Exhibit  4.  Print/Design  Specifications  for 
"Choosing  Your  Medicaid  Health  Plan" 
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Photoshop  version  4.0 
Aldus  Freehand  version  7.0 

Adobe  Stone  Sans  (for  cover  and  subhead  text) 
Benguiat  (for  headings  in  text  and  chart) 
Adobe  Goudy  (for  body  and  chart  text) 

The  cover  photo  and  photos  on  pages  1,2,4,5,6 
and  7  were  purchased  from  Photodisc  for  use  in  this 
handbook.  Photos  on  pages  3  and  10  were 
borrowed  from  a  personal  collection  and  can  only 
be  used  in  this  publication. 

Black  and  PMS  485  (red) 

The  screened  boxes  on  pages  2,  4,  6-13, 

and  the  back  cover  are  created  with  an  overprint  of  a 

10%  black  box  over  a  10%  PMS  485  box  which  then 

produces  a  third  'tan'  color. 


Please  note:  The  fonts  and  photos  used  in  this  handbook  are  provided  on  the 
disc,  however,  their  use  is  restricted  to  printing  of  the  handbooks  for  this  project 
(HCFA  contract  #500-94-0048).  They  can  be  purchased  for  use  in  other  projects 
through  Adobe  (fonts)  and  Photodisc  (photos). 
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Exhibit  4.  Print/Design  Specifications  for 
"Choosing  A  Medicare  HMO"  (continued) 

Computer  Format:  Macintosh 

Programs:  QuarkXPress  version  3.31  Page  Layout  Program 

Photoshop  version  4.0 
Aldus  Freehand  version  7.0 

Adobe  Caxton  (for  cover,  subhead,  body  and  chart  text) 
Helvetica  (for  cover) 

The  cover  photo  and  photos  on  pages  l,3,4,and  12  were 
purchased  from  Photodisc  for  use  in  this  handbook. 
Photos  on  pages  2  and  6-9  were  borrowed  from  a 
personal  collection  and  can  only  be  used  in  this 
publication. 

Paper  texture  used  on  the  cover  and  charts  on 
pages  13-15  is  a  Photoshop  tiff  file  and  included 
on  the  disc. 

Ink  Colors:  Black  and  PMS  321  (teal  blue) 

Please  note:  The  fonts  and  photos  used  in  this  handbook  are  provided  on  the 
disc,  however,  their  use  is  restricted  to  printing  of  the  handbooks  for  this  project 
(HCFA  contract  #500-94-0048).  They  can  be  purchased  for  use  in  other  projects 
through  Adobe  (fonts)  and  Photodisc  (photos). 


Fonts: 


Photos: 


Texture: 
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Exhibit  4.  Print/Design  Specifications  for 
"Medicare  Options"  (continued) 


Computer  Format:  Macintosh 


Programs: 


Fonts: 


Photos: 


Ink  Colors: 


QuarkXPress  version  3.31  Page  Layout  Program 
Photoshop  version  4.0 
Aldus  Freehand  version  7.0 

Adobe  Stone  Sans  (for  cover,  subheads  and  chart  text) 
Adobe  Caxton  (for  body  and  chart  text) 

Cover  photo  was  purchased  from  Photodisc  for  use 
in  this  handbook. 

PMS  492  (red)  and  PMS  540  (dark  blue) 


Please  note:  The  fonts  and  photos  used  in  this  handbook  are  provided  on  the  disc, 
however,  their  use  is  restricted  to  printing  of  the  handbooks  for  this  project  (HCFA 
contract  #500-94-0048).  They  can  be  purchased  for  use  in  other  projects  through 
Adobe  (fonts)  and  Photodisc  (photos). 
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Chapter  5:  Data  Issues 


Data  issues  will  no  doubt  be  one  of  the  most  significant  issues  for  developing 
materials  to  guide  health  care  choices  and  decisionmaking.  This  chapter  discusses  two 
types  of  data  used  in  the  prototype  handbooks:  health  plan  performance  data,  which  are 
used  in  the  Medicare  and  Medicaid  HMO  handbooks,  and  estimates  of  average  out-of- 
pocket  expenses  incurred  by  high,  medium,  and  low  users  of  health  care,  which  are  used  in 
the  Medicare  Options  handbook.  The  discussion  of  performance  measures  addresses 
several  important  issues  concerning  the  choice  of  measures  and  format  for  the  prototypes 
and  the  availability  and  quality  of  performance  data  in  general,  which  users  of  these 
prototypes  should  keep  in  mind  when  adapting  them  for  practical  applications.  The 
discussion  of  out-of-pocket  expense  estimates  describes  the  rationale  for  providing  these 
data  and  the  methods  used  to  compute  them. 

5.1     Performance  Measures 

5.1.1  Choice  of  Performance  Measures 

Measuring  the  performance  of  HMOs  and  similar  network-based  health  plans  is  a 
new  field  and  it  is  evolving  rapidly.  At  the  beginning  of  this  project,  in  October  1 994,  there 
was  no  consensus  about  which  measures  were  meaningful,  how  the  data  should  be 
collected,  and  how  they  should  be  reported.  Since  then,  the  state-of-the-art  has  advanced 
significantly.  The  NCQA  has  developed  HEDIS  3.0,  which  includes  measures  for  both  the 
privately  and  publicly  insured  (Medicare  and  Medicaid  beneficiaries),  and  AHCPR's 
Consumer  Assessments  of  Health  Plans  Study  (CAHPS)  has  released  initial  versions  of 
consumer  assessment  surveys  for  the  Medicare  and  Medicaid  populations.  Despite  this 
design  work,  however,  there  has  been  little  implementation  of  performance  measurement 
at  the  HMO  level  for  the  Medicare  and  Medicaid  populations.  Thus,  these  prototype 
handbooks  were  developed  without  specific  sources  of  data  to  support  them.  The  effect  of 
this  situation  is  that  the  measures  included  in  the  prototype  handbooks  are  placeholders 
rather  than  specific  recommendations.  They  were  selected  to  include  measures  that  we 
believe  are  relevant  and  meaningful  to  the  Medicare  and  Medicaid  populations,  based  on 
our  cognitive  research.  However,  it  is  not  very  useful  to  specify  measures  of  individual 
variables  or  composite  aggregations  until  we  know  what  data  will  be  available. 

Each  organization  in  each  jurisdiction  that  uses  a  performance  measurement  tool 
will  have  to  consider  which  measures  from  that  tool  should  be  reported.  Given  the 
emergence  of  CAHPS  and  HEDIS  as  the  leading  measurement  tools,  users  of  our 
prototype  report  formats  may  want  to  rely  on  the  recommendations  from  those  two  sources 
regarding  which  performance  measures  to  report.  In  choosing  measures,  they  will  have  to 
address  several  important  issues  in  addition  to  the  basic  question  about  which  measures  to 
present: 

•       Medical  records  or  survey-based  consumer  assessments?  Research 
has  suggested  that  beneficiaries  find  records-based  utilization  measures, 
such  as  the  percentage  of  current  beneficiaries  who  receive  selected 
services,  less  informative  for  their  choice  of  health  plan  than  survey-based 
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consumer  assessments.  Whether  they  are  less  relevant  or  simply  less 
familiar  and  less  well-understood  remains  to  be  determined.  If  the  latter,  it 
suggests  that  records-based  utilization  measures  will  become  more  valuable 
as  beneficiaries  become  more  familiar  with  them.  Thus,  our  prototype  reports 
include  measures  of  both  types. 

Should  utilization  measures  related  to  ambulatory  sensitive  conditions 
be  included? 

Ambulatory  sensitive  conditions  (ASCs)  are  those  that  should  not  usually 
result  in  hospitalization  if  they  are  treated  appropriately  in  ambulatory  care 
settings.  Thus,  low  hospitalization  rates  for  ASCs  have  been  interpreted  to 
indicate  high  quality  care  in  health  care  delivery  systems.  Common  examples 
include  hospitalizations  for  diabetes,  pneumonia,  and  pediatric  asthma. 

The  use  of  ASC  hospitalizations  to  represent  quality  is  appropriate  in  a  fee- 
for-service  (FFS)  system  because  providers  benefit  financially  from  high 
utilization  rates.  In  capitated  systems  such  as  Medicare  and  Medicaid  HMOs, 
however,  the  interpretation  is  not  as  clear.  Providers  in  these  systems  benefit 
financially  by  discouraging  hospitalizations.  Thus,  it  is  not  clear  if  low  ASC 
hospitalization  rates  reflect  high  quality  care  or  barriers  to  access  in  response 
to  financial  incentives.  Focus  group  participants  recognized  this  ambiguity. 
To  avoid  confusion  we  recommend  that  ASC  hospitalization  rates  not  be 
included  among  performance  measures  for  beneficiaries. 

How  many  measures  should  be  provided?  The  question  of  which 
measures  to  include  is  complicated  by  a  significant  constraint  on  the  number 
that  can  be  provided  and  used  effectively.  There  are  obvious  constraints 
concerning  the  overall  burden  and  the  cost  associated  with  large  reports,  so 
there  are  real  incentives  to  keep  it  small.  More  importantly,  there  are 
constraints  on  the  cognitive  ability  of  beneficiaries  to  interpret  complex, 
unfamiliar  data.  The  cognitive  demands  increase  significantly  with  the 
number  of  measures,  which  can  be  a  particularly  significant  problem  for  aged 
Medicare  beneficiaries  or  Medicaid  beneficiaries  with  less-than-average 
education. 

It  seems  likely  that  beneficiaries'  ability  to  interpret  these  measures  correctly 
will  improve  with  time  and  familiarity.  The  prototypes  use  few  measures 
(eight)  under  the  assumption  that  beneficiaries'  interest  in  and  ability  to  use 
them  will  increase  with  time  and  that  more,  and  more  complex,  measures 
should  be  added  in  response  to  the  demand  for  more  information  from 
beneficiaries. 

Should  the  measures  vary?  The  limit  on  the  number  of  measures  that 
beneficiaries  can  use  effectively  means  that  choices  will  have  to  be  made. 
One  way  to  expand  the  number  of  measures  that  can  be  presented  is  to  vary 
them  over  time.  This  strategy  also  minimizes  the  likelihood  that  the  ability  of 
a  measure  to  discriminate  among  HMOs  will  decrease  with  time  as  HMOs 
focus  their  quality  control  efforts  on  measures  that  are  reported  to 
beneficiaries. 
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5.1.2  Presentation  of  Performance  Measures 

Our  focus  groups  indicated  that  graphs  were  easier  for  beneficiaries  to  understand 
than  tables.  The  Medicare  and  Medicaid  HMO  prototype  handbooks  both  use  vertical  bar 
graphs  that  report  a  single  number  or  percentage  for  each  HMO.  The  graphs  are  preceded 
by  two  examples  that  explain  to  beneficiaries  how  they  should  be  read.  Next  to  each  graph 
is  a  paragraph  that  explains  the  meaning  of  the  measure,  whether  a  higher  or  lower 
number  is  better,  and,  except  for  the  excellent-to-poor  ratings  where  we  believe  it  is 
obvious,  why  a  higher  or  lower  number  is  better.  These  graphs  worked  well  with  both 
Medicare  and  Medicaid  beneficiaries  in  cognitive  testing.  Beneficiaries  were  almost  always 
able  to  identify  the  best-performing  HMO  on  each  measure. 

For  excellent-to-poor  ratings,  the  bar  represents  the  percentage  reporting  the  two 
highest  categories  on  the  five-point  scale,  excellent  and  very  good.  Research  by  others 
has  suggested  that  beneficiaries  prefer  to  see  the  entire  distribution  included  in  the  bar. 
This  approach  provides  beneficiaries  with  much  more  information,  but  it  also  increases  the 
cognitive  burden  of  interpretation.  In  view  of  our  concern  about  minimizing  the  cognitive 
burden  of  performance  measures  for  Medicare  and  Medicaid  beneficiaries,  we  have 
included  only  one  component  of  the  distribution  in  the  bar  graphs. 

The  graphs  present  data  for  four  HMOs.  The  cognitive  burden  of  making 
comparisons  increases  with  the  number  of  HMOs  included  in  the  graphs.  The  complexity 
and  cost  of  design  and  production  of  the  reports  also  increase  with  the  number  of  HMOs, 
particularly  if  the  number  of  HMOs  exceeds  the  number  of  bars  that  can  be  included  in  a 
graph  with  sufficient  font  size  on  a  single  page.  Users  of  these  prototypes  will  have  to 
adapt  the  graphs  if  more  than  four  HMOs  are  being  compared.  Testing  of  graphs  with  data 
for  more  HMOs  is  a  potential  objective  for  future  cognitive  testing. 

The  placement  of  the  performance  measures  in  the  handbook  is  also  an  important 
presentation  issue.  In  these  prototypes,  the  measures  come  before  the  descriptive 
information  about  covered  services  and  premiums.  For  the  nine  Medicare  beneficiaries  in 
our  final  round  of  cognitive  testing,  we  compared  their  statements  about  which  of  the 
performance  measures  were  most  valuable  to  them  with  how  well  their  chosen  HMO 
performed  on  those  measures.  Only  a  small  number  picked  HMOs  that  performed  best  on 
the  measures  they  said  were  most  valuable.  The  bene.iciaries  reported  that,  in  general, 
they  found  the  performance  measures  useful,  so  we  do  not  believe  that  they  disregarded 
the  information.  Instead,  they  may  have  relied  on  the  benefits  and  cost  information  more 
than  on  the  performance  measures.  The  benefits  and  cost  information  could  be  inherently 
more  salient  or  it  could  have  been  more  influential  in  this  test  because  they  were  the  last 
data  the  beneficiaries  received  before  they  made  their  choice.  Future  cognitive  research 
on  the  presentation  of  performance  measures  should  vary  the  order  in  which  performance 
measures  and  descriptive  benefits  and  cost  information  are  placed  in  the  handbooks  to 
determine  if  placement  is  related  to  salience. 

Threats  to  Validity.  Performance  measures  are  intended  to  discriminate  among 
HMOs  in  a  specific  locale.  Small  differences  between  two  HMOs  can  be  attributed  to  error 
or  bias  of  one  kind  or  another  rather  than  to  meaningful  differences.  Even  when  a 
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difference  is  not  attributable  to  error  or  bias,  it  may  be  so  small  that  it  has  little  practical  or 
clinical  meaning.  Beneficiaries  who  participated  in  our  focus  groups  were  aware  in  general 
that  error  and  bias  can  affect  the  validity  of  the  reported  differences.  Thus,  we  concluded 
that  it  is  important  to  provide  them  with  some  guidance  about  how  error  and  bias  might 
affect  the  results  as  part  of  the  information  we  provide  about  interpreting  the  measures. 
Three  potential  threats  to  validity  are  discussed  below. 

Conflict  of  Interest  Beneficiaries  who  participated  in  our  focus  groups  were 
concerned  that  data  collected  by  the  HMOs  being  evaluated  could  be  biased 
to  reflect  positively  on  the  HMOs.  Because  of  this  concern,  we  recommend 
that  data  be  collected  by  the  Medicare  program  and  state  Medicaid  programs, 
or  that  data  collection  be  conducted  by  community-wide  coalitions  of 
purchasers,  including  employers  and  small  business  purchasing  cooperatives 
as  well  as  the  Medicaid  and  Medicare  programs.  To  assure  beneficiaries  that 
the  performance  measures  were  collected  by  an  objective  party,  we  included 
a  statement  that  the  measures  were  collected  fairly  and  accurately.  In 
cognitive  testing,  this  statement  seemed  to  be  a  sufficient  assurance  for 
beneficiaries.  Although  it  is  a  very  general  statement,  it  can  be  used  with  any 
system  of  data  collection.  Once  systems  of  data  collection  are  established, 
the  purchasers  who  adapt  these  prototypes  to  their  locales  should  consider 
more  specific  statements  about  who  collected  the  data  and  how  they  were 
obtained. 

Selection  Bias.  There  is  a  general  consensus  that  records-based 
performance  measures  of  use  and  outcomes  should  be  adjusted  to  reflect 
variation  in  need  for  health  care  among  the  enrolled  populations  of  the  HMOs 
being  compared.  The  argument  applies  also  to  survey-based  consumer 
assessments,  although  there  is  a  significant  counter-argument  that 
unadjusted  satisfaction  measures  are  more  relevant  because  health  plans 
should  satisfy  members  in  the  areas  of  access  and  communication  regardless 
of  their  health  status  and  need  for  care.  The  final  decision  about  whether  to 
adjust  for  risk  depends  very  much  on  the  data  that  are  available  in  each  local 
market  and  whether  the  performance  measurement  tools  used  in  each 
market  include  data  that  are  useful  for  adjustment. 

An  alternative  to  adjusting  the  aggregate  population  measures  for  risk 
selection  is  to  report  performance  measures  separately  for  high  users  of 
health  care  or  persons  with  significant  chronic  illness.  The  way  in  which 
capitated  health  plans  treat  very  ill  beneficiaries  is  a  potentially  important 
indicator  of  performance  for  healthy  persons  as  well  as  those  who  are  ill 
because  of  the  financial  incentive  to  minimize  services  under  capitation. 
However,  separate  estimates  for  high  users  cannot  be  made  from  survey 
data,  and  from  medical  records  also  if  samples  are  used,  unless  the  initial 
samples  include  a  sufficiently  large  number  of  persons  in  this  relatively  small 
group.  Thus,  the  decision  to  report  separate  measures  for  high  users  has 
serious  implications  for  sampling  design  and  the  cost  of  collecting  data.  It 
also  expands  the  number  of  measures  being  reported,  which,  as  noted 
above,  creates  additional  cognitive  burden  for  persons  who  use  the 
measures.  We  have  not  specified  risk-adjusted  measures  or  separate 
measures  for  high  users  in  the  prototypes  because  of  these  problems  and 
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uncertainties.  However,  it  is  an  important  issue  on  which  standards  will 
continue  to  evolve.  Local  users  who  adapt  these  prototypes  should  pay 
attention  to  the  recommendations  of  the  designers  of  the  performance 
measurement  tools  they  use  (e.g.,  CAHPS,  HEDIS). 

Sampling  Error.  Survey-based  porformance  measures,  and,  in  many  cases, 
records-based  measures  as  well,  are  obtained  from  representative, 
probability  samples.  Thus,  there  is  almost  always  sampling  variance 
associated  with  each  reported  measure.  In  cognitive  testing,  we  asked 
beneficiaries  whether  they  perceived  small  differences  between  HMOs  as 
meaningful  and  found  that  some  do  and  others  do  not.  We  concluded  that  it 
is  important  to  inform  beneficiaries  who  are  not  familiar  with  the  concept  that 
small  differences  may  not  be  meaningful  and  to  provide  guidance  to  all 
beneficiaries  about  which  differences  are  not  significant. 

Sampling  variance  is  a  difficult  concept  to  convey  to  persons  who  are  not 
familiar  with  statistics.  It  is  particularly  difficult  in  this  context  because  we 
have  little  space  available  and  because  the  size  of  the  confidence  interval 
around  each  estimate  will  vary  based  on  variation  in  sample  size  and  the 
proportion  reported.  Sample  sizes  may  vary  by  HMO  and  by  the  performance 
measure  (e.g.,  some  measures  may  be  reported  only  for  women  and  others 
for  women  and  men  combined).  Given  the  space  limitations,  reporting 
separate  confidence  intervals  for  each  measure  is  not  feasible.  We  reviewed 
attempts  in  a  few  HMO  performance  reports  and  found  them  too  technical. 
Thus,  for  the  final  prototypes,  we  have  included  a  statement  that  the  margin 
of  error  for  each  estimate  is  about  10  percent  and  provided  an  example  of 
how  two  estimates  less  than  10  percent  apart  are  probably  not  meaningfully 
different,  but  two  others  more  than  10  percent  apart  probably  are  different. 
The  margin  of  error  approach  is  used  often  in  the  mass  media  when  reporting 
public  opinion  survey  data.  Thus,  we  believe  that  it  is  the  approach  most 
likely  to  be  recognized  by  Medicare  and  Medicaid  beneficiaries. 

The  choice  of  1 0  percent  for  the  prototypes  is  arbitrary.  The  number  used  in 
actual  applications  must  be  derived  from  the  performance  measurement  data. 
It  should  probably  be  an  average  (e.g.,  mean,  median,  mode)  half-width  95 
percent  confidence  interval  for  all  the  measures  reported. 

Sponsorship  of  Data  Collection  and  Reporting.  The  responsibility  for  collecting 
and  reporting  data  about  Medicare  HMOs  could  reside  at  the  federal  level,  with  HCFA  or  at 
the  local  level  with  community  coalitions  or  Professional  Review  Organizations  (PROs). 
The  responsibility  for  Medicaid  HMOs  appears  to  reside  with  the  states.  But  as  yet  there  is 
no  structure  in  place  at  any  level  to  collect  or  report  HMO  performance  measures.  The 
most  significant  advance  in  this  regard  is  simply  the  development  of  the  CAHPS  and 
HEDIS  measurement  tools  for  Medicare  and  Medicaid  populations.  The  next  most 
significant  advance  is  HCFA's  contract  for  a  national  survey  of  Medicare  HMOs,  using  the 
Medicare  CAHPS  instrument.  Several  states  (e.g.  Florida,  Oregon,  Minnesota)  have  either 
collected  performance  measures  from  their  Medicaid  populations  or  are  actively  engaged  in 
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planning  for  data  collection,  but  they  represent  a  minority  of  states.  And,  in  some  cases, 
performance  data  have  been  used  for  program  management  but  have  not  been  reported  to 
beneficiaries  facing  a  choice  among  HMOs. 

HMOs  are  increasingly  concerned  about  multiple  requests  for  data  from  the 
Medicare  and  Medicaid  programs,  employers,  and  purchasing  cooperatives.  Because 
HMO  performance  reports  must  focus  on  the  specific  markets  or  locales  in  which  HMO 
provider  networks  operate,  a  community-based  data  collection  and  reporting  structure  is 
very  appealing.  It  would  minimize  the  burden  on  HMOs  and  would  provide  for  consistent 
sampling  and  data  collection  methods  among  all  HMOs  and  beneficiary  populations. 

Developing  these  prototypes  in  the  absence  of  a  structure  for  data  collection  and 
reporting  has  been  a  challenge  because  there  are  so  many  unknowns.  It  will  be  important 
for  users  of  these  prototypes  to  modify  them  to  reflect  the  actual  sponsors  and  methods  of 
data  collection  in  their  locales. 

5.2    Market-Specific  Variation  of  Cost  and  Benefit  Data  in  the  Prototypes 

In  order  to  make  the  handbooks  market-specific,  selected  benefit  and  cost  data  will 
need  to  be  revised  before  using.  In  the  Medicaid  and  Medicare  HMO  handbooks,  HMO- 
specific  premium  and  benefit  cost  information  must  be  inserted  into  the  comparison  charts. 

The  worksheet  in  the  Medicare  Options  handbook  requires  the  most  revisions.  One 
area  affected  by  market-specific  variation  is  the  Medigap  section  of  the  worksheets  which 
was  designed  to  be  used  in  conjunction  with  a  state-specific  Medigap  premium  guide.9 
Another  area  that  may  require  modification  is  the  list  of  the  standard  benefits  covered  by 
the  10  Medigap  plans  approved  as  part  of  the  Omnibus  Budget  Reconciliation  Act  of  1990 
(OBRA  90).  Because  three  states  (Massachusetts,  Minnesota,  and  Wisconsin)  are  exempt 
from  requiring  the  10  standard  policies,  developers  in  these  state  will  need  to  revise  the 
handbook  to  reflect  their  own  standard  policies.  Another  issue  that  may  affect  the  Medigap 
benefit  charts  is  the  potential  to  sell  standard  plans  with  an  innovative  benefit  as  permitted 
under  OBRA  90.  Although  few  innovative  benefits  have  been  approved,  they  exist  in  some 
areas.  Supplementary  text  may  also  be  needed  to  explain  these  state-  or  market-specific 
nuances. 

In  every  location,  developers  would  need  to  insert  market-specific  HMO  data  that 
reflect  the  plans  in  their  region.  In  regions  where  no  HMOs  exist,  the  handbook  would 
require  more  extensive  changes  to  account  for  this.  However,  given  the  rapid  growth  in  the 
number  of  Medicare  HMOs,  this  will  become  less  of  an  issue. 

On  page  9,  we  provide  a  chart  comparing  the  Medicare  and  Medigap  option  with 
Medicare  HMOs  across  the  following  dimensions:  choice  of  providers,  access  to  speciality 


9  Readers  are  told  to  obtain  the  premium  guide  before  beginning  the  worksheet  section.  Such 
guides  are  currently  available  in  at  least  40  states.  In  states  where  no  premium  guide  exists,  consumers 
will  need  to  contact  individual  insurance  companies,  agents,  or  brokers,  to  obtain  plan-level  premium 
data  required  to  complete  the  cost  worksheets. 
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care,  and  geographic  availability  of  services,  benefits,  and  premiums.  Market-specific 
developers  will  need  to  insert  the  range  of  premiums  available  in  their  area.  Similarly,  on 
page  1 1 ,  developers  will  need  to  insert  the  state's  income  eligibility  level  for  Medicaid  to 
assist  readers  in  determining  whether  they  might  qualify  as  a  Medicare/Medicaid  dual 
eligible.  Of  course,  each  handbook  will  need  to  be  periodically  updated  to  reflect  market 
changes  (e.g.,  entry  and  exit  of  specific  health  plans,  changes  in  the  Medicare  program, 
and  other  factors  affected  by  health  care  reform). 

5.3    Out-of  Pocket  Expenses  for  Medicare  Options 

Use  of  Out-of-Pocket  Expenses  in  the  Worksheet.  The  worksheets  in  the 
Medicare  Options  handbook  include  estimates  of  out-of-pocket  expenses  that  beneficiaries 
would  incur,  on  average,  under  each  Medicare  supplemental  insurance  plan  and  Medicare 
HMO  described  in  the  handbook.  Our  intent  was  to  provide  beneficiaries  with  a  way  to 
estimate  their  total  expenses  under  each  option,  by  adding  their  expected  out-of-pocket 
costs  to  the  premium  for  the  specific  products  they  are  considering  to  arrive  at  a  total 
expected  expense  for  each  competing  benefit  package.  This  information  is  important  if 
beneficiaries  are  to  be  able  to  make  informed  benefit-cost  comparisons. 

It  is  very  difficult  to  make  this  information  understandable.  The  range  of  options 
available  to  Medicare  beneficiaries  is  quite  broad,  especially  for  new  beneficiaries  who 
have  been  insured  through  employment-related  groups  and  never  had  to  select  from  more 
than  two  or  three  options  at  most,  or  who  have  rarely  had  health  insurance.  Beneficiaries 
are  usually  confused  about  the  scope  of  benefits  offered  by,  and  the  rules  for  using, 
increasingly  complex  managed  care  plans.  Adding  cost  information  significantly  expands 
the  cognitive  burden. 

Computation  of  Out-of-Pocket  Expenses.  The  computation  of  expected  out-of- 
pocket  expenses  had  four  steps,  which  are  summarized  below  but  are  explained  in  greater 
detail  in  Appendix  A.  The  first  step,  based  on  the  method  used  by  Garfinkel,  Riley,  and 
lannacchione  (1988)  to  estimate  the  characteristics  of  high-cost  users  of  health  care,  used 
the  1987  National  Medical  Expenditure  Survey  (NMES)  data  to  identify  Medicare 
beneficiaries  who  used  less  than  their  proportional  share  of  total  medical  expenditures  (low 
users),  about  their  share  of  total  expenditures  (medium  users),  and  more  than  their  share 
of  total  expenditures  (high  users). 

The  second  step  involved  computing  the  mean  number  of  health  services  of  each 
type  reported  in  NMES  (e.g.,  hospital  stays,  physician  visits,  prescriptions  filled)  for  each  of 
the  three  groups  of  beneficiaries.  These  utilization  profiles  are  included  in  the  handbook  to 
assist  beneficiaries  in  assigning  themselves  to  a  category. 

The  third  step  involved  computing  the  mean  expenditure  for  each  type  of  service  in 
each  of  the  three  expense  categories.  The  fourth  step,  based  in  general  on  the  method 
used  by  Sofaer,  Kenney,  and  Davidson  (1992),  involved  the  application  of  either  Medicare 
and  Medigap  benefits  or  Medicare  HMO  benefits  to  the  mean  expenditures  for  each  type  of 
service  to  estimate  the  residual  out-of-pocket  expenditure  after  the  various  insurance 
options  pay  benefits.  Our  methodology  is  described  in  more  detail  in  Appendix  A. 
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If  these  out-of-pocket  expense  estimates  are  to  be  useful  they  will  have  to  be 
available  to  local  publishers  of  the  Medicare  Options  handbook.  In  the  absence  of  a 
dissemination  plan,  we  cannot  prescribe  a  specific  approach,  but  there  are  several 
important  considerations.  The  out-of-pocket  expense  estimates  are  affected  by  four  basic 
factors:  (1)  definition  of  the  high,  medium,  and  low  user  categories;  (2)  the  mean 
expenditures  by  type  of  service  within  each  user  category;  (3)  medical  price  inflation;  and 
(4)  the  application  of  Medigap  and  Medicare  HMO  benefits  to  those  means.  Some  of  these 
elements  will  vary  annually  but  others  will  not. 

The  first  two  elements,  definition  of  user  categories  and  mean  expenditures  by  type 
of  service  within  category,  are  based  on  the  1987  National  Medical  Expenditure  Survey. 
We  used  NMES  as  the  source  of  data  because  it  provides  detailed  information  about 
expenditures  by  type  of  service  and  it  provides  data  about  services  that  are  covered  by 
Medigap  policies  and  Medicare  HMOs  but  are  not  covered  by  Medicare.  The  only 
alternative  source  of  data,  Medicare  claims  data,  do  not  include  data  on  services  not 
covered  by  Medicare,  especially  prescription  drugs,  and  so  are  not  useful  for  computing 
residual  out-of-pocket  expenses  after  Medigap  and  Medicare  HMO  payments.  The 
periodic  nationwide  surveys  of  medical  expenditures  conducted  by  the  Agency  for  Health 
Care  Policy  and  Research  are  the  only  source  of  data  with  sufficient  detail  for  this  purpose. 
Because  these  surveys  are  conducted  only  about  once  each  decade,  annual  data  on  the 
underlying  distributions  used  to  define  high-,  medium-,  and  low-cost  users  and  the  mean 
expenditures  by  type  of  service  are  not  available.  Thus,  it  is  not  possible  to  vary  these  two 
components  of  the  out-of-pocket  expense  estimates  annually.  However,  it  would  be 
desirable  to  recompute  the  use  categories  and  expenditure  estimates  whenever  the  data 
for  a  new  national  expenditure  survey  are  released. 

Medical  prices  increase  annually,  but  the  estimates  can  be  easily  adjusted  by 
applying  a  medical  price  inflator.  For  the  prototype,  we  adjusted  the  1987  NMES  estimates 
to  1995  using  the  medical  component  of  the  Consumer  Price  Index. 

Because  of  standardization  of  Medigap  plans,  Medigap  benefits  will  not  vary 
annually  or  by  locale.  Thus,  the  set  of  out-of-pocket  expense  estimates  for  the  10  standard 
Medigap  plans  provided  in  the  prototype  can  be  used  by  all  local  publishers  (except  in  the 
three  waiver  states,  where  residual  expenses  would  have  to  be  computed  separately  for 
their  local  plans).  Greater  precision  could  be  obtained  by  computing  regional  estimates  of 
mean  expenditures  by  type  of  service  and  user  category,  but  that  seems  unnecessary, 
particularly  until  this  worksheet  becomes  widely  used.  Medicare  HMO  benefits,  however, 
vary  by  HMO,  making  the  task  of  computing  the  residual  out-of-pocket  expense  much  more 
complex.  Separate  computations  would  have  to  be  made  for  each  HMO  that  offers  unique 
benefits. 

Although  the  method  of  computing  residual  out-of-pocket  expenses  is  complex,  it 
could  be  done  locally.  Nevertheless,  to  assure  methodological  consistency,  minimize  local 
burden,  and  increase  the  likelihood  that  the  estimates  would  be  used  by  local  publishers, 
HCFA  should  produce  the  estimates  annually  for  Medigap  plans  A  through  J,  for  the  plans 
in  the  three  waiver  states,  and  for  each  Medicare  HMO.  The  computations  would  be  most 
complex  whenever  data  from  a  new  national  medical  expenditure  survey  become  available, 
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but  they  would  be  relatively  simple  in  other  years  when  only  medical  price  inflation  and 
HMO  benefit  changes  have  to  be  considered. 
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Chapter  6.0:  Lessons  Learned  from  Materials  Development  and  Testing 


In  this  final  chapter,  we  provide  a  brief  overview  of  the  major  lessons  learned  from 
the  project,  highlighting  those  learned  during  the  materials  development  and  testing 
phases. 

6.1  Education  and  Information  Needs  of  Consumers 

Our  data  suggest  that  many  eligible  individuals  have  limited  knowledge  of  the 
Medicare  and  Medicaid  programs.  In  fact,  some  Medicare  and  Medicaid  beneficiaries  do 
not  even  realize  that  they  have  choices  to  make  about  how  they  receive  their  health  care 
services.  These  choices  can  significantly  affect  the  quality  of,  access  to,  and  cost  of  care 
they  receive,  and  are  becoming  increasingly  important  with  the  movement  toward  managed 
care  plans  that  lock  beneficiaries  in  for  a  defined  period  of  time.  Without  beneficiaries 
having  knowledge  of  the  basic  health  insurance  concepts  and  the  major  trade-offs 
associated  with  different  plan  choices,  successfully  conveying  information  about  quality  of 
care  differences  across  health  plans  poses  significant  challenges.  Furthrr,  our  data 
suggest  that  the  choice  process  is  complex  and  is  most  appropriately  made  through  the 
capacity  to  synthesize  multiple  variables  within  a  personal  frame  of  reference.  Thus,  the 
process  of  providing  beneficiaries  with  information  must  be  coupled  with  an  educational 
component  to  provide  sufficient  context  and  background  for  understanding  the  material 
before  beginning  the  process  of  plan  by  plan  comparisons.  The  beneficiaries  in  our  focus 
groups  and  cognitive  testing  interviews  expressed  a  keen  interest  in  both  the  educational 
and  comparative  components  of  the  health  plan  materials  we  developed  and  many 
indicated  that  they  would  use  such  materials  if  they  were  made  available  to  them. 

6.2  Format  and  Content  of  Materials 

In  order  for  beneficiaries  to  reap  the  benefits  of  health  plan  materials,  the  materials 
must  be  designed  carefully  and  contain  the  appropriate  blend  of  information.  One  of  the 
greatest  challenges  in  developing  comparative  health  plan  materials  is  identifying  the 
appropriate  level  of  detail  to  provide  without  overwhelming  or  confusing  the  reader.  Difficult 
decisions  must  be  made  about  what  material  will  be  included  and  excluded  and  how  it  will 
be  arranged  in  order  to  minimize  cognitive  burden  and  maximize  comprehension. 

"Layering"  information  is  one  approach  that  can  help  to  address  the  dilemma  about 
the  amount  of  information  to  present  to  consumers.  Layering  involves  presenting  the  same 
information  in  two  or  three  different  ways,  each  with  progressively  more  detail.  The  more 
detailed  version  of  the  information  might  be  presented  in  an  appendix  or  a  separate  piece 
of  materials.  The  approach  can  appease  both  types  of  users:  those  who  want  a  minimal 
amount  of  information  and  those  who  want  more  detail.  One  potential  pitfall  of  layering, 
however,  is  the  possibility  that  the  detailed  layer  and  the  summary  layer  might  convey 
inconsistent  or  even  conflicting  messages  to  the  reader.  This  may  be  a  problem  for 
performance  measures,  in  particular,  if  aggregate  measures  suggest  there  is  no  difference 
between  health  plans  but  more  disaggregated  layers  of  data  indicate  there  is  a  difference. 
Thus,  the  use  of  layering  must  take  the  potential  for  conflicting  messages  into  account. 
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The  use  of  navigational  aids  such  as  introductory  instructions  or  section  overviews 
were  essential  for  guiding  the  user  through  either  printed  or  electronic  materials  and  we 
strongly  recommend  that  these  be  included  in  all  consumer  materials.  These  often 
included  visual  aids  such  as  a  finger  with  a  string  tied  around  it  to  remind  the  reader  that 
definitions  for  boldfaced  terms  could  be  found  in  the  glossary  of  the  handbook.  Although 
simple,  these  aids  can  help  consumers  gather  information  and  come  to  a  decision. 

We  also  learned  that  consumers  favored  having  some  form  of  an  algorithm  for 
making  the  health  plan  decision.  The  worksheet  in  the  Medicare  Options  handbook  was 
specifically  designed  with  this  purpose  in  mind.  It  identifies  a  step  by  step  process  and 
guides  the  user  through  the  decision  process.  This  approaches  breaks  down  the  cognitive 
task  into  a  set  of  smaller,  more  manageable  tasks,  but  does  not  exclude  any  important 
features  that  should  be  considered  in  the  decision. 

Providing  beneficiaries  with  all  the  information  they  need  to  choose  a  health  plan 
usually  means  increasing  the  amount  and  complexity  of  information  presented.  This 
increase  threatens  to  provide  beneficiaries  with  more  information  than  they  can  manage. 
The  trade-off  between  "information  overload"  and  providing  useful  information  encourages 
designers  to  compensate  for  necessarily  complex  or  extensive  sections  of  the  material  by 
simplifying  in  other  places.  We  learned  from  cognitive  testing,  however,  that  including 
material  that  in  itself  might  be  unnecessary  can  have  beneficial  consequences,  as  long  as 
testing  indicates  that  it  does  not  overload  users.  The  most  striking  example  of  this 
approach  concerned  the  benefits  comparison  charts  for  Medigap  plans  and  HMOs  in  the 
Medicare  Options  book. 

Because  the  benefits  comparison  for  all  1 0  standard  Medigap  plans  completely  uses 
both  facing  pages  in  the  book,  HMOs  appear  in  a  separate  chart.  Thus,  beneficiaries  who 
try  to  compare  Medigap  overall  to  HMOs  overall  have  to  flip  back  and  forth,  which  is  a 
significant  navigation  problem  according  to  focus  groups  and  cognitive  tests.  The  testing 
revealed  several  techniques  to  make  this  comparison  easier,  some  of  which  add  material 
that  might  otherwise  be  considered  unnecessary: 

•  Use  the  same  matrix  format  for  both  charts:  health  plans  in  columns 
and  benefits  in  rows. 

•  Use  the  same  benefits  categories  (i.e.,  rows)  in  both  charts,  even  if  the 
information  does  not  vary  within  one  of  the  charts.  For  example, 
dental  care  is  not  covered  by  any  of  the  standard  Medigap  plans,  but 
limited  dental  benefits  are  provided  by  some  HMOs.  Excluding  dental 
care  from  the  Medigap  chart  because  coverage  does  not  vary  would 
be  an  obvious  way  to  simplify  a  complex,  dense  chart.  But  the 
presence  of  dental  care  in  the  HMO  chart  will  cause  beneficiaries  to 
wonder  about  how  Medigap  plans  treat  dental  benefits,  creating 
confusion.  Through  cognitive  testing,  we  determined  that  any 
increased  confusion  that  might  result  from  keeping  an  "unnecessary" 
category  is  more  than  offset  by  the  decrease  that  comes  from  making 
the  comparison  between  Medigap  plans  and  HMOs  easier. 
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•  The  information  about  benefits  covered  by  each  Medigap  plan  should 
be  the  sum  of  coverage  from  both  Medicare  and  the  Medigap  plan. 
The  main  reason  for  taking  this  approach  is  that  HMOs  provide  a 
single  set  of  benefits  through  one  insurer.  It  is  very  difficult  for 
beneficiaries  to  compare  this  single  set  of  integrated  benefits  with  two 
sets  if  Medicare  benefits  and  Medigap  benefits  are  presented 
separately.  This  is  another  example  of  how  making  the  comparison 
chart  more  complex  simplifies  the  beneficiary's  task. 

•  Including  benefits  categories  that  do  not  vary  for  either  HMOs  or 
Medigap  plans  in  the  comparison  charts  can  be  a  very  effective  way  to 
convey  information  about  services  that  are  never  covered.  Focus 
groups  and  cognitive  testing  confirmed  that  Medicare  beneficiaries, 
especially  new  or  about-to-be  new  beneficiaries,  are  confused  about 
what  Medicare  covers.  Many  think  that  they  will  have  few  if  any  out-of- 
pocket  expenses  after  they  become  eligible  for  Medicare.  Although 
long-term  care  is  not  covered  by  Medicare,  Medigap  plans,  or  HMOs, 
we  included  it  in  both  the  Medigap  and  the  HMO  comparison  charts 
because  it  makes  it  clear  to  beneficiaries  that  it  is  important  to  consider 
long-term  care  and  that  none  of  these  options  covers  it.  The  long-term 
care  category  contributes  no  information  about  the  coverage  provided 
by  any  of  the  options,  so  it  could  easily  be  dropped  to  simplify  the 
charts.  But  it  provides  information  about  the  equally  important  issue  of 
what  is  not  covered. 

These  design  strategies  increase  the  complexity  and  density  of  the  comparison 
charts  and  the  information  they  include.  However,  through  cognitive  testing,  we 
determined  that  the  additional  information  was  valuable  to  beneficiaries  in  clarifying  their 
choices  and  did  not  exceed  the  amount  of  information  that  they  could  manage. 

Ultimately,  the  contents  of  the  comparative  plan  materials  will  be  driven  by  what 
reliable  and  valid  data  are  available  for  use  at  the  time  of  development.  Significant 
progress  is  being  made  with  respect  to  the  refinement  of  both  survey-based  and  clinical 
records-based  performance  measures,  both  in  terms  of  how  they  should  be  defined  and 
collected.  Health  plan  materials  will  need  to  evolve  as  the  research  community  makes 
advances  in  these  areas. 

Regardless  of  what  information  is  included  in  the  materials,  an  overall  goal  should 
be  education  of  the  consumer.  Simply  providing  information  to  consumers  about  their 
health  plan  choices  will  not  be  sufficient.  Consumers  need  to  be  educated  about  how  the 
Medicaid,  Medicare,  or  other  health  insurance  systems  "work"  and  what  that  means  to 
them  as  an  individual.  One  specific  area  where  consumers  certainly  need  educational 
assistance  is  with  interpretation  of  performance  measures. 

6.3     Evaluation  of  Consumer  Materials 

This  project  highlighted  the  importance  of  consumer  testing  throughout  the  process 
of  materials  development.  Although  focus  groups  were  helpful  in  the  initial  phases  of  the 
project  for  exploring  consumers'  preferences  for  information  and  presentation  styles,  we 
consider  cognitive  testing  to  be  the  definitive  methodology  for  empirically  assessing  the 
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usefulness  of  materials  once  developed.  Despite  the  fact  that  the  materials  had  been 
reviewed  by  relevant  staff  from  HCFA  as  well  as  our  technical  advisory  panel  prior  to 
testing,  the  cognitive  testing  phase  revealed  points  for  misinterpretation  or  problems  in  use 
by  consumers  that  would  have  diminished  the  value  of  the  materials  if  not  addressed.  The 
one-to-one  interactive  nature  of  cognitive  testing  allowed  us  to  systematically  assess  input 
from  each  individual  participant  through  observing  the  consumer  using  the  materials, 
asking  specific  questions  about  comprehension  and  preferences,  and  exploring  areas  of 
confusion.  These  advantages  contrast  to  the  necessity  of  relying  on  the  "group  level"  of 
performance  as  is  the  case  for  focus  groups.  We  encourage  the  empirical  evaluation  of 
materials  through  a  systematic,  consumer-based  methodology  like  cognitive  testing  as  an 
integral  part  of  future  materials  development  activities. 

6.4    Dissemination  of  Consumer  Materials  and  the  Role  of  Intermediaries 

Numerous  decisions  need  to  be  made  about  the  most  effective  and  cost-efficient 
methods  for  disseminating  consumer  health  plan  materials.  We  learned  that  different  types 
of  consumers  prefer  to  receive  materials  in  different  ways,  but  many  favored  a  combination 
of  written  materials  that  they  could  review  on  their  own  in  conjunction  with  a  counselor  to 
whom  they  could  address  questions  either  over  the  telephone  or  in  person.  Several 
different  types  of  intermediaries  -  enrollment  brokers,  benefit  managers,  ICA  counselors  - 
could  potentially  play  a  role  in  the  information  dissemination  process.  Regardless  of  who 
does  become  involved,  some  long-range  planning  and  training  of  staff  will  be  essential. 

The  Balanced  Budget  Act  of  1 997  calls  for  the  dissemination  of  information  about 
health  plan  choices  to  all  new  and  current  Medicare  beneficiaries  beginning  in  1998. 
Specifically,  it  requires  that  the  Secretary  of  Health  and  Human  Services  disseminate 
general  information  about  benefits,  election  procedures,  beneficiary  rights,  HMOs, 
Medigap,  MSAs,  and  SELECT,  and  the  potential  for  contract  termination,  as  well  as 
comparative  information  about  benefits,  cost  sharing,  networks,  coverage  protocols  for 
emergency  care,  premiums,  service  area,  and  quality  and  performance  (if  available).  This 
groundbreaking  legislation  will  certainly  initiate  the  development  and  distribution  of 
information  to  Medicare  beneficiaries,  but  many  questions  remain  unanswered  about  the 
process  for  carrying  out  the  legislative  mandate.  In  addition,  several  state  Medicaid 
programs  that  have  turned  to  mandatory  managed  care  programs  are  showing  interest  in 
distributing  information  to  beneficiaries  who  must  choose  between  competing  plans  in  their 
area. 

This  project  represents  one  of  the  pioneering  efforts  to  develop  comparative  health 
plan  information  for  consumers.  It  stands  among  the  earliest  attempts  to  identify  consumer 
preferences  for  information  content  and  presentation  of  health  care  choices  and  then  to 
actually  test  the  effectiveness  of  the  materials  for  consumer  consumption  and  use.  Clearly, 
the  field  has  evolved  since  the  project  began  in  late  1994.  Our  hope  is  that  the  products 
and  approaches  developed  through  this  project  will  serve  as  a  foundation  for  future 
research  to  guide  consumers  in  making  well-informed,  appropriate  choices  for  health  care 
coverage. 
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